Jun 11 20:25:34.529236 ip-10-0-137-254 systemd[1]: Starting Kubernetes Kubelet... Jun 11 20:25:34.925912 ip-10-0-137-254 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 11 20:25:34.925912 ip-10-0-137-254 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jun 11 20:25:34.925912 ip-10-0-137-254 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 11 20:25:34.925912 ip-10-0-137-254 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jun 11 20:25:34.925912 ip-10-0-137-254 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 11 20:25:34.926615 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.925995 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jun 11 20:25:34.932471 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932452 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 11 20:25:34.932471 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932470 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932475 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932479 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932482 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932485 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932488 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932491 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932494 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932496 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932499 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932502 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932505 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932507 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932510 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932513 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932516 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932518 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932521 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932529 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932532 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 11 20:25:34.932545 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932535 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932537 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932540 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932542 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932545 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932547 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932550 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932553 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932558 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932562 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932565 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932568 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932571 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932573 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932576 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932578 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932581 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932583 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932586 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 11 20:25:34.933072 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932588 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932591 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932593 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932595 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932598 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932600 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932603 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932605 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932608 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932611 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932614 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932617 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932619 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932622 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932627 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932630 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932633 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932635 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932638 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932641 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 11 20:25:34.933524 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932643 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932646 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932648 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932651 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932654 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932657 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932660 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932662 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932665 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932667 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932670 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932673 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932675 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932678 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932681 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932683 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932686 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932688 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932691 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932693 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 11 20:25:34.934037 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932696 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932698 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932701 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932703 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932706 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932709 2564 feature_gate.go:328] unrecognized feature gate: Example Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932713 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932718 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932722 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932726 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932729 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932732 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932735 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932737 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932740 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932743 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932745 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932748 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.932751 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 11 20:25:34.934514 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933467 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933474 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933477 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933481 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933484 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933487 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933490 2564 feature_gate.go:328] unrecognized feature gate: Example Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933493 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933496 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933500 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933504 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933507 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933509 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933512 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933514 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933517 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933519 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933523 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933526 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 11 20:25:34.934992 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933529 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933531 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933534 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933537 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933539 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933542 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933545 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933547 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933550 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933552 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933555 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933557 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933559 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933562 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933565 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933567 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933570 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933572 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933575 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933578 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 11 20:25:34.935460 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933581 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933583 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933586 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933588 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933591 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933593 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933595 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933599 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933601 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933604 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933606 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933609 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933612 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933615 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933617 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933620 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933622 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933624 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933628 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933630 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 11 20:25:34.935998 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933633 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933635 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933640 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933643 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933646 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933648 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933651 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933654 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933656 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933658 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933661 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933663 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933666 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933669 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933671 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933674 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933676 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933679 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933681 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933684 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 11 20:25:34.936557 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933686 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933689 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933692 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933695 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933700 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933703 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933706 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933709 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933712 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933714 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933717 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933720 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933722 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933725 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933728 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933730 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933733 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933735 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933738 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 11 20:25:34.937137 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.933740 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934169 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934175 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934179 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934182 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934185 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934187 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934190 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934193 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934195 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934198 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934201 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934204 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934206 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934209 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934212 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934217 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934221 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934223 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934226 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 11 20:25:34.937653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934229 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934231 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934234 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934236 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934239 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934241 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934244 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934246 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934249 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934251 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934254 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934256 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934259 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934262 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934265 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934267 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934270 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934272 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934275 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 11 20:25:34.938227 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934278 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934280 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934283 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934285 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934288 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934290 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934293 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934297 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934300 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934304 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934307 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934309 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934312 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934315 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934317 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934322 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934326 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934329 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934332 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 11 20:25:34.938794 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934335 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934338 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934341 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934344 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934346 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934349 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934351 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934354 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934357 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934359 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934362 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934364 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934366 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934369 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934372 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934375 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934377 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934380 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934382 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934385 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 11 20:25:34.939323 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934390 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934393 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934398 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934401 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934404 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934407 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934410 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934412 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934415 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934418 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934420 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934423 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934426 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934428 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934431 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934433 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934436 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934438 2564 feature_gate.go:328] unrecognized feature gate: Example Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934441 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 11 20:25:34.939875 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934443 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934445 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.934448 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934514 2564 flags.go:64] FLAG: --address="0.0.0.0" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934527 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934535 2564 flags.go:64] FLAG: --anonymous-auth="true" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934541 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934549 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934553 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934560 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934564 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934568 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934571 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934576 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934579 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934583 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934586 2564 flags.go:64] FLAG: --cgroup-root="" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934589 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934592 2564 flags.go:64] FLAG: --client-ca-file="" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934595 2564 flags.go:64] FLAG: --cloud-provider="external" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934598 2564 flags.go:64] FLAG: --cluster-dns="[]" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934603 2564 flags.go:64] FLAG: --cluster-domain="" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934606 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jun 11 20:25:34.940418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934609 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934612 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934615 2564 flags.go:64] FLAG: --container-log-max-files="5" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934619 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934622 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934626 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934629 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934632 2564 flags.go:64] FLAG: --contention-profiling="false" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934635 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934638 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934641 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934644 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934648 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934652 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934654 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934657 2564 flags.go:64] FLAG: --enable-load-reader="false" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934660 2564 flags.go:64] FLAG: --enable-server="true" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934664 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934668 2564 flags.go:64] FLAG: --event-burst="100" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934671 2564 flags.go:64] FLAG: --event-qps="50" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934674 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934677 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934683 2564 flags.go:64] FLAG: --eviction-hard="" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934687 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jun 11 20:25:34.941024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934690 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934693 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934696 2564 flags.go:64] FLAG: --eviction-soft="" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934699 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934702 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934706 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934709 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934712 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934714 2564 flags.go:64] FLAG: --fail-swap-on="true" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934717 2564 flags.go:64] FLAG: --feature-gates="" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934721 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934724 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934727 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934730 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934733 2564 flags.go:64] FLAG: --healthz-port="10248" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934736 2564 flags.go:64] FLAG: --help="false" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934739 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-137-254.ec2.internal" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934742 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934745 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934748 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934751 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934755 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934758 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jun 11 20:25:34.941622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934761 2564 flags.go:64] FLAG: --image-service-endpoint="" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934763 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934766 2564 flags.go:64] FLAG: --kube-api-burst="100" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934770 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934773 2564 flags.go:64] FLAG: --kube-api-qps="50" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934776 2564 flags.go:64] FLAG: --kube-reserved="" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934779 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934782 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934786 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934789 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934792 2564 flags.go:64] FLAG: --lock-file="" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934795 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934798 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934801 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934807 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934810 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934812 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934815 2564 flags.go:64] FLAG: --logging-format="text" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934818 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934821 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934824 2564 flags.go:64] FLAG: --manifest-url="" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934827 2564 flags.go:64] FLAG: --manifest-url-header="" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934831 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934834 2564 flags.go:64] FLAG: --max-open-files="1000000" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934839 2564 flags.go:64] FLAG: --max-pods="110" Jun 11 20:25:34.942226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934842 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934844 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934847 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934850 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934853 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934856 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934859 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934866 2564 flags.go:64] FLAG: --node-status-max-images="50" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934869 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934872 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934875 2564 flags.go:64] FLAG: --pod-cidr="" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934878 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a2e90c8ef7d783e4d635e24fb09435f5e1f6e954583512c0944f9e57bc77b52" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934884 2564 flags.go:64] FLAG: --pod-manifest-path="" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934887 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934890 2564 flags.go:64] FLAG: --pods-per-core="0" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934894 2564 flags.go:64] FLAG: --port="10250" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934897 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934900 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0569f1f3ddb7df0e1" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934903 2564 flags.go:64] FLAG: --qos-reserved="" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934906 2564 flags.go:64] FLAG: --read-only-port="10255" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934909 2564 flags.go:64] FLAG: --register-node="true" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934912 2564 flags.go:64] FLAG: --register-with-taints="" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934916 2564 flags.go:64] FLAG: --registry-burst="10" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934919 2564 flags.go:64] FLAG: --registry-qps="5" Jun 11 20:25:34.942832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934921 2564 flags.go:64] FLAG: --reserved-cpus="" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934924 2564 flags.go:64] FLAG: --reserved-memory="" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934928 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934931 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934934 2564 flags.go:64] FLAG: --rotate-certificates="false" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934937 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934940 2564 flags.go:64] FLAG: --runonce="false" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934943 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934946 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934949 2564 flags.go:64] FLAG: --seccomp-default="false" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934951 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934954 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934957 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934982 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934985 2564 flags.go:64] FLAG: --storage-driver-password="root" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934988 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934991 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934994 2564 flags.go:64] FLAG: --storage-driver-user="root" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.934998 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935001 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935005 2564 flags.go:64] FLAG: --system-cgroups="" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935008 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935014 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935017 2564 flags.go:64] FLAG: --tls-cert-file="" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935021 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jun 11 20:25:34.943458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935026 2564 flags.go:64] FLAG: --tls-min-version="" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935029 2564 flags.go:64] FLAG: --tls-private-key-file="" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935032 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935035 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935038 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935040 2564 flags.go:64] FLAG: --v="2" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935048 2564 flags.go:64] FLAG: --version="false" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935052 2564 flags.go:64] FLAG: --vmodule="" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935056 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935059 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935173 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935177 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935180 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935184 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935187 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935189 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935192 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935194 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935197 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935200 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935202 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935205 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 11 20:25:34.944108 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935207 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935211 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935214 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935217 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935220 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935222 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935225 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935228 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935232 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935236 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935239 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935241 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935244 2564 feature_gate.go:328] unrecognized feature gate: Example Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935247 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935249 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935252 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935254 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935257 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935260 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 11 20:25:34.944641 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935262 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935264 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935267 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935269 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935272 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935274 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935277 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935279 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935281 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935284 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935286 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935289 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935291 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935294 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935296 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935299 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935301 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935304 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935306 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935309 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 11 20:25:34.945117 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935312 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935315 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935319 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935322 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935324 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935327 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935329 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935332 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935334 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935337 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935341 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935344 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935347 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935350 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935353 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935355 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935358 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935360 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935363 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 11 20:25:34.945623 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935366 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935368 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935371 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935373 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935376 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935378 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935381 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935383 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935386 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935388 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935391 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935395 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935398 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935401 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935407 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935411 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935414 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935417 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935420 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 11 20:25:34.946114 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935422 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935425 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935428 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935431 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935433 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935436 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935438 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935441 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935443 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.935446 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.935451 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.942198 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.942214 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.942233 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.942239 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jun 11 20:25:34.946570 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942295 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942300 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942303 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942306 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942309 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942312 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942315 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942317 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942320 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942324 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942328 2564 feature_gate.go:328] unrecognized feature gate: Example Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942332 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942336 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942339 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942342 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942345 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942347 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942350 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942353 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 11 20:25:34.947064 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942356 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942358 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942361 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942363 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942367 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942371 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942374 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942377 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942379 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942382 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942386 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942389 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942391 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942394 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942396 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942399 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942401 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942404 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942408 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 11 20:25:34.947535 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942411 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942414 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942417 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942420 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942423 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942425 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942428 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942431 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942434 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942436 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942439 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942441 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942444 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942446 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942449 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942451 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942454 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942456 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942458 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942461 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 11 20:25:34.948009 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942463 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942466 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942469 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942472 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942474 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942477 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942480 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942482 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942485 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942488 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942491 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942493 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942496 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942498 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942500 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942503 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942506 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942508 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942510 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 11 20:25:34.948498 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942513 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942516 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942518 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942521 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942524 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942526 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942529 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942532 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942534 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942536 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942539 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942542 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942544 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942547 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942550 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942552 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942555 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942558 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942561 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942563 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 11 20:25:34.948976 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942566 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942568 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.942573 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942686 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942691 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942694 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942697 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942699 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942702 2564 feature_gate.go:328] unrecognized feature gate: Example Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942704 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942707 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942709 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942712 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942714 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942718 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 11 20:25:34.949497 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942720 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942723 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942726 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942728 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942731 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942734 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942736 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942739 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942742 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942744 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942746 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942749 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942751 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942754 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942757 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942759 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942762 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942764 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942767 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942769 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 11 20:25:34.949859 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942772 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942774 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942778 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942782 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942784 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942787 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942790 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942793 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942795 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942798 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942800 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942803 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942806 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942808 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942812 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942814 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942816 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942820 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942822 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 11 20:25:34.950364 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942825 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942827 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942830 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942832 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942835 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942838 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942841 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942844 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942846 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942849 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942852 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942854 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942858 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942861 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942864 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942867 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942870 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942873 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942876 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 11 20:25:34.950845 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942879 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942881 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942884 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942887 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942889 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942893 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942896 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942899 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942902 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942904 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942907 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942910 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942912 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942915 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942917 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942919 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942922 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942925 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942927 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 11 20:25:34.951519 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942934 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942936 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942939 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942941 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942944 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942946 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942949 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942952 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942955 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:34.942957 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.942976 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.943090 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.946899 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.948340 2564 server.go:1019] "Starting client certificate rotation" Jun 11 20:25:34.952002 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.948432 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 11 20:25:34.952412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.948464 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 11 20:25:34.952412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.950673 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 11 20:25:34.953746 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.953718 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 11 20:25:34.963163 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.963144 2564 log.go:25] "Validated CRI v1 runtime API" Jun 11 20:25:34.967718 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.967696 2564 log.go:25] "Validated CRI v1 image API" Jun 11 20:25:34.968529 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.968512 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jun 11 20:25:34.971741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.971720 2564 fs.go:135] Filesystem UUIDs: map[67700291-3f7f-4d31-b88b-e912cf62a4eb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 876b1482-380f-421b-8945-e87c5723d6dd:/dev/nvme0n1p4] Jun 11 20:25:34.971830 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.971740 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jun 11 20:25:34.976623 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.976460 2564 manager.go:216] Machine: {Timestamp:2026-06-11 20:25:34.975911197 +0000 UTC m=+0.330657504 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100540 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec278a95c437933bc7d45ecb2703ebb0 SystemUUID:ec278a95-c437-933b-c7d4-5ecb2703ebb0 BootID:84cd6f2d-8c9d-4073-a14a-d8221b2b6ee2 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:33:09:e9:bc:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:33:09:e9:bc:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:42:fc:dc:fb:9b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jun 11 20:25:34.976721 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.976615 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jun 11 20:25:34.976779 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.976767 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.120.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260608-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jun 11 20:25:34.980017 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.979979 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jun 11 20:25:34.980165 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.980019 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-254.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jun 11 20:25:34.980921 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.980910 2564 topology_manager.go:138] "Creating topology manager with none policy" Jun 11 20:25:34.980977 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.980923 2564 container_manager_linux.go:309] "Creating device plugin manager" Jun 11 20:25:34.980977 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.980931 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 11 20:25:34.981607 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.981594 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 11 20:25:34.981809 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.981791 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 11 20:25:34.982556 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.982545 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jun 11 20:25:34.985251 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.985237 2564 state_mem.go:36] "Initialized new in-memory state store" Jun 11 20:25:34.985372 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.985362 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jun 11 20:25:34.985403 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.985393 2564 kubelet.go:486] "Attempting to sync node with API server" Jun 11 20:25:34.985440 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.985404 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jun 11 20:25:34.985440 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.985420 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jun 11 20:25:34.985440 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.985432 2564 kubelet.go:398] "Adding apiserver pod source" Jun 11 20:25:34.985440 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.985441 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jun 11 20:25:34.985773 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.985763 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 11 20:25:34.985821 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.985784 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 11 20:25:34.987832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.987813 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jun 11 20:25:34.989227 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.989213 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jun 11 20:25:34.989276 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.989244 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jun 11 20:25:34.990917 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990903 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jun 11 20:25:34.990975 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990924 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jun 11 20:25:34.990975 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990932 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jun 11 20:25:34.990975 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990937 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jun 11 20:25:34.990975 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990946 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jun 11 20:25:34.990975 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990952 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jun 11 20:25:34.990975 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990974 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jun 11 20:25:34.991134 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990981 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jun 11 20:25:34.991134 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990989 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jun 11 20:25:34.991134 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.990994 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jun 11 20:25:34.991134 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.991008 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jun 11 20:25:34.991134 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.991014 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jun 11 20:25:34.991707 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.991696 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jun 11 20:25:34.991707 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.991706 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jun 11 20:25:34.995725 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.995708 2564 server.go:1268] "Started kubelet" Jun 11 20:25:34.995901 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.995845 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jun 11 20:25:34.995985 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.995936 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jun 11 20:25:34.996093 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.995868 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jun 11 20:25:34.996743 ip-10-0-137-254 systemd[1]: Started Kubernetes Kubelet. Jun 11 20:25:34.997138 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:34.997112 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jun 11 20:25:34.997232 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:34.997112 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jun 11 20:25:34.997232 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.997226 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-254.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jun 11 20:25:34.997867 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.997469 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jun 11 20:25:34.998167 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:34.998155 2564 server.go:310] "Adding debug handlers to kubelet server" Jun 11 20:25:35.002640 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.002623 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jun 11 20:25:35.002640 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.002634 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jun 11 20:25:35.002786 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.002717 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.003296 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.003239 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jun 11 20:25:35.003457 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.003439 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jun 11 20:25:35.003521 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.003440 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jun 11 20:25:35.003521 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.003471 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jun 11 20:25:35.003595 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.003578 2564 reconstruct.go:97] "Volume reconstruction finished" Jun 11 20:25:35.003595 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.003587 2564 reconciler.go:29] "Reconciler: start to sync state" Jun 11 20:25:35.003680 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.003657 2564 factory.go:55] Registering systemd factory Jun 11 20:25:35.003680 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.003677 2564 factory.go:223] Registration of the systemd container factory successfully Jun 11 20:25:35.004000 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.003984 2564 factory.go:153] Registering CRI-O factory Jun 11 20:25:35.004000 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.004003 2564 factory.go:223] Registration of the crio container factory successfully Jun 11 20:25:35.004165 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.004102 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jun 11 20:25:35.004165 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.004127 2564 factory.go:103] Registering Raw factory Jun 11 20:25:35.004165 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.004142 2564 manager.go:1196] Started watching for new ooms in manager Jun 11 20:25:35.004481 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.004467 2564 manager.go:349] Starting recovery of all containers Jun 11 20:25:35.004905 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.003989 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-254.ec2.internal.18b820a283f4aeb9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-254.ec2.internal,UID:ip-10-0-137-254.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-254.ec2.internal,},FirstTimestamp:2026-06-11 20:25:34.995672761 +0000 UTC m=+0.350419067,LastTimestamp:2026-06-11 20:25:34.995672761 +0000 UTC m=+0.350419067,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-254.ec2.internal,}" Jun 11 20:25:35.008753 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.008627 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jun 11 20:25:35.008753 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.008646 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jun 11 20:25:35.014459 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.014435 2564 manager.go:354] Recovery completed Jun 11 20:25:35.019940 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.019926 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 11 20:25:35.021841 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.021822 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientMemory" Jun 11 20:25:35.021902 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.021854 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasNoDiskPressure" Jun 11 20:25:35.021902 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.021864 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientPID" Jun 11 20:25:35.022442 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.022428 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jun 11 20:25:35.022506 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.022442 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jun 11 20:25:35.022506 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.022447 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 11 20:25:35.022506 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.022461 2564 state_mem.go:36] "Initialized new in-memory state store" Jun 11 20:25:35.023847 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.023747 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-254.ec2.internal.18b820a28583fa23 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-254.ec2.internal,UID:ip-10-0-137-254.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-254.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-254.ec2.internal,},FirstTimestamp:2026-06-11 20:25:35.021840931 +0000 UTC m=+0.376587238,LastTimestamp:2026-06-11 20:25:35.021840931 +0000 UTC m=+0.376587238,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-254.ec2.internal,}" Jun 11 20:25:35.023917 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.023873 2564 policy_none.go:49] "None policy: Start" Jun 11 20:25:35.023917 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.023890 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jun 11 20:25:35.023917 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.023904 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jun 11 20:25:35.024039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.023992 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientMemory" Jun 11 20:25:35.024039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.024015 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasNoDiskPressure" Jun 11 20:25:35.024039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.024030 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientPID" Jun 11 20:25:35.024872 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.024862 2564 policy_none.go:47] "Start" Jun 11 20:25:35.030009 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.029990 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxmd9" Jun 11 20:25:35.034219 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.034143 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-254.ec2.internal.18b820a285843fcb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-254.ec2.internal,UID:ip-10-0-137-254.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-254.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-254.ec2.internal,},FirstTimestamp:2026-06-11 20:25:35.021858763 +0000 UTC m=+0.376605070,LastTimestamp:2026-06-11 20:25:35.021858763 +0000 UTC m=+0.376605070,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-254.ec2.internal,}" Jun 11 20:25:35.035396 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.035378 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxmd9" Jun 11 20:25:35.058651 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.058633 2564 manager.go:337] "Starting Device Plugin manager" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.058677 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.058691 2564 server.go:87] "Starting device plugin registration server" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.059001 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.059015 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.059100 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.059198 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.059208 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.061034 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.061085 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.068491 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.069638 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.069677 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.069697 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jun 11 20:25:35.070374 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.069734 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jun 11 20:25:35.071776 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.071753 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 11 20:25:35.159289 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.159251 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 11 20:25:35.160506 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.160487 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientMemory" Jun 11 20:25:35.160635 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.160523 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasNoDiskPressure" Jun 11 20:25:35.160635 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.160538 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientPID" Jun 11 20:25:35.160635 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.160582 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.170543 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.170520 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.170672 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.170547 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-254.ec2.internal\": node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.170672 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.170623 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal"] Jun 11 20:25:35.170755 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.170687 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 11 20:25:35.172229 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.172213 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientMemory" Jun 11 20:25:35.172321 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.172243 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasNoDiskPressure" Jun 11 20:25:35.172321 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.172253 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientPID" Jun 11 20:25:35.173309 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.173297 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 11 20:25:35.173435 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.173421 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.173497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.173453 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 11 20:25:35.174075 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.174056 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientMemory" Jun 11 20:25:35.174184 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.174088 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasNoDiskPressure" Jun 11 20:25:35.174184 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.174101 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientPID" Jun 11 20:25:35.174184 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.174102 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientMemory" Jun 11 20:25:35.174184 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.174125 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasNoDiskPressure" Jun 11 20:25:35.174184 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.174141 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientPID" Jun 11 20:25:35.175227 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.175212 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.175303 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.175240 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 11 20:25:35.175882 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.175866 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientMemory" Jun 11 20:25:35.176005 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.175893 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasNoDiskPressure" Jun 11 20:25:35.176005 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.175904 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientPID" Jun 11 20:25:35.191782 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.191762 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.195639 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.195623 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-254.ec2.internal\" not found" node="ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.199362 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.199347 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-254.ec2.internal\" not found" node="ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.292300 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.292268 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.305622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.305593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a24d161ecf306ac01e33da9ffb33262b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal\" (UID: \"a24d161ecf306ac01e33da9ffb33262b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.305695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.305625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a24d161ecf306ac01e33da9ffb33262b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal\" (UID: \"a24d161ecf306ac01e33da9ffb33262b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.305695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.305641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99904d988681166720269c6ecc3484aa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-254.ec2.internal\" (UID: \"99904d988681166720269c6ecc3484aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.392767 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.392731 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.406148 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.406121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a24d161ecf306ac01e33da9ffb33262b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal\" (UID: \"a24d161ecf306ac01e33da9ffb33262b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.406223 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.406151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a24d161ecf306ac01e33da9ffb33262b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal\" (UID: \"a24d161ecf306ac01e33da9ffb33262b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.406223 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.406168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99904d988681166720269c6ecc3484aa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-254.ec2.internal\" (UID: \"99904d988681166720269c6ecc3484aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.406223 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.406195 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99904d988681166720269c6ecc3484aa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-254.ec2.internal\" (UID: \"99904d988681166720269c6ecc3484aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.406356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.406240 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a24d161ecf306ac01e33da9ffb33262b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal\" (UID: \"a24d161ecf306ac01e33da9ffb33262b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.406356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.406285 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a24d161ecf306ac01e33da9ffb33262b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal\" (UID: \"a24d161ecf306ac01e33da9ffb33262b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.493640 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.493567 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.498718 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.498704 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.502738 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.502716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal" Jun 11 20:25:35.594080 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.594030 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.694625 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.694594 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.795195 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.795106 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.895714 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.895676 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:35.948215 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.948177 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jun 11 20:25:35.948726 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:35.948351 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jun 11 20:25:35.996704 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:35.996670 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:36.031264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.031076 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jun 11 20:25:36.040101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.040065 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-10 20:20:35 +0000 UTC" deadline="2027-12-26 10:12:03.747999939 +0000 UTC" Jun 11 20:25:36.040101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.040096 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13501h46m27.70790749s" Jun 11 20:25:36.086225 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:36.086177 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99904d988681166720269c6ecc3484aa.slice/crio-ef34214bc9b1bc5fc182921267aea99bb63ed421243768e9542ee8885fa13d44 WatchSource:0}: Error finding container ef34214bc9b1bc5fc182921267aea99bb63ed421243768e9542ee8885fa13d44: Status 404 returned error can't find the container with id ef34214bc9b1bc5fc182921267aea99bb63ed421243768e9542ee8885fa13d44 Jun 11 20:25:36.086822 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:36.086798 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24d161ecf306ac01e33da9ffb33262b.slice/crio-5d0c2c98a08c27282ba4116812d0576eccf23e613fa4838fd10b4e1bf74150d5 WatchSource:0}: Error finding container 5d0c2c98a08c27282ba4116812d0576eccf23e613fa4838fd10b4e1bf74150d5: Status 404 returned error can't find the container with id 5d0c2c98a08c27282ba4116812d0576eccf23e613fa4838fd10b4e1bf74150d5 Jun 11 20:25:36.090945 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.090927 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 11 20:25:36.097320 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:36.097301 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:36.135099 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.135065 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 11 20:25:36.136184 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.136167 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientMemory" Jun 11 20:25:36.136300 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.136197 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasNoDiskPressure" Jun 11 20:25:36.136300 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.136208 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeHasSufficientPID" Jun 11 20:25:36.197840 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:36.197804 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:36.298402 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:36.298361 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-254.ec2.internal\" not found" Jun 11 20:25:36.314458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.314362 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jun 11 20:25:36.403600 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.403522 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" Jun 11 20:25:36.415581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.415559 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 11 20:25:36.417217 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.417203 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal" Jun 11 20:25:36.424694 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.424676 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 11 20:25:36.475246 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.475220 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jun 11 20:25:36.986874 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.986842 2564 apiserver.go:52] "Watching apiserver" Jun 11 20:25:36.996113 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.996081 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jun 11 20:25:36.996487 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.996460 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-v8w2z","kube-system/konnectivity-agent-r28zn","openshift-multus/multus-additional-cni-plugins-97sxc","openshift-multus/multus-v8xfv","openshift-multus/network-metrics-daemon-wqxvp","openshift-network-diagnostics/network-check-target-jv82t","openshift-network-operator/iptables-alerter-fk57z","kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6","openshift-cluster-node-tuning-operator/tuned-qw9mb","openshift-dns/node-resolver-5fnqb","openshift-image-registry/node-ca-k58bv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-x5z7f"] Jun 11 20:25:36.998100 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.998005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:36.999031 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:36.999012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.000742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.000336 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jun 11 20:25:37.000742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.000385 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8dj9k\"" Jun 11 20:25:37.000742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.000554 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jun 11 20:25:37.000742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.000607 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jun 11 20:25:37.000742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.000698 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.001145 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.001044 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jun 11 20:25:37.001145 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.001112 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzkr4\"" Jun 11 20:25:37.001256 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.001183 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jun 11 20:25:37.002087 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.002069 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.003303 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.003233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:37.003395 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.003234 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jun 11 20:25:37.003521 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.003321 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:37.003958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.003771 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jun 11 20:25:37.003958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.003827 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jun 11 20:25:37.003958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.003851 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jun 11 20:25:37.004157 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.004137 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jun 11 20:25:37.004297 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.004276 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jun 11 20:25:37.004373 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.004304 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jun 11 20:25:37.004444 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.004430 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d84tc\"" Jun 11 20:25:37.004687 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.004671 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:37.004767 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.004732 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:37.005038 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.005023 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2vqcb\"" Jun 11 20:25:37.006569 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.006096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.008838 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.008815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.008934 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.008912 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jun 11 20:25:37.009075 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.009056 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jun 11 20:25:37.009559 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.009532 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4hfxf\"" Jun 11 20:25:37.010654 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.010605 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jun 11 20:25:37.012227 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.011936 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jun 11 20:25:37.012227 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.012046 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jun 11 20:25:37.012227 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.012077 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:37.012412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.012276 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jun 11 20:25:37.012412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.012279 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jun 11 20:25:37.012724 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.012702 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mqxv4\"" Jun 11 20:25:37.014371 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.014350 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jun 11 20:25:37.014462 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.014385 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jun 11 20:25:37.014644 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.014589 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.015040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.015022 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jun 11 20:25:37.015278 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.015260 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cm4tx\"" Jun 11 20:25:37.015633 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.015588 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jun 11 20:25:37.015748 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.015731 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.015829 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.015807 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:37.016664 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/942c56c3-9856-4d26-82a4-f9d7c1adc1f9-iptables-alerter-script\") pod \"iptables-alerter-fk57z\" (UID: \"942c56c3-9856-4d26-82a4-f9d7c1adc1f9\") " pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.016736 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-cnibin\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.016736 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-run-netns\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.016736 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-var-lib-cni-multus\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.016736 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-var-lib-kubelet\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.016912 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f748b10a-2034-4330-9292-f4781f1b4359-tmp-dir\") pod \"node-resolver-5fnqb\" (UID: \"f748b10a-2034-4330-9292-f4781f1b4359\") " pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.016912 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f748b10a-2034-4330-9292-f4781f1b4359-hosts-file\") pod \"node-resolver-5fnqb\" (UID: \"f748b10a-2034-4330-9292-f4781f1b4359\") " pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.016912 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7plss\" (UniqueName: \"kubernetes.io/projected/9f84094c-d325-462d-8f1b-a71c48ad5524-kube-api-access-7plss\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.016912 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016812 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jun 11 20:25:37.016912 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-socket-dir-parent\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.016912 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016873 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xlnbr\"" Jun 11 20:25:37.016912 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/942c56c3-9856-4d26-82a4-f9d7c1adc1f9-host-slash\") pod \"iptables-alerter-fk57z\" (UID: \"942c56c3-9856-4d26-82a4-f9d7c1adc1f9\") " pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f84094c-d325-462d-8f1b-a71c48ad5524-cni-binary-copy\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016935 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.016951 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f84094c-d325-462d-8f1b-a71c48ad5524-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-system-cni-dir\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-os-release\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8xv\" (UniqueName: \"kubernetes.io/projected/942c56c3-9856-4d26-82a4-f9d7c1adc1f9-kube-api-access-zg8xv\") pod \"iptables-alerter-fk57z\" (UID: \"942c56c3-9856-4d26-82a4-f9d7c1adc1f9\") " pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-daemon-config\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.017231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69a4d19c-0866-442f-af12-6bd23fc4c17e-cni-binary-copy\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snrj\" (UniqueName: \"kubernetes.io/projected/86b6d922-1a55-4844-b308-127fc062e056-kube-api-access-8snrj\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-tuning-conf-dir\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-run-k8s-cni-cncf-io\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-conf-dir\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-run-multus-certs\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017453 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-etc-kubernetes\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017477 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-os-release\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-var-lib-cni-bin\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9q9w\" (UniqueName: \"kubernetes.io/projected/69a4d19c-0866-442f-af12-6bd23fc4c17e-kube-api-access-r9q9w\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpr9x\" (UniqueName: \"kubernetes.io/projected/f748b10a-2034-4330-9292-f4781f1b4359-kube-api-access-qpr9x\") pod \"node-resolver-5fnqb\" (UID: \"f748b10a-2034-4330-9292-f4781f1b4359\") " pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-system-cni-dir\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f84094c-d325-462d-8f1b-a71c48ad5524-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.017691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-cni-dir\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.018434 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-cnibin\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.018434 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.017756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-hostroot\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.019354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.019334 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jun 11 20:25:37.019354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.019347 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jun 11 20:25:37.019481 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.019370 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nq2dj\"" Jun 11 20:25:37.019481 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.019446 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jun 11 20:25:37.019662 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.019638 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 11 20:25:37.053115 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.053088 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqzbc" Jun 11 20:25:37.063182 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.063156 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqzbc" Jun 11 20:25:37.075283 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.075239 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal" event={"ID":"99904d988681166720269c6ecc3484aa","Type":"ContainerStarted","Data":"ef34214bc9b1bc5fc182921267aea99bb63ed421243768e9542ee8885fa13d44"} Jun 11 20:25:37.077238 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.077201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" event={"ID":"a24d161ecf306ac01e33da9ffb33262b","Type":"ContainerStarted","Data":"5d0c2c98a08c27282ba4116812d0576eccf23e613fa4838fd10b4e1bf74150d5"} Jun 11 20:25:37.104690 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.104660 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jun 11 20:25:37.118230 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-sysconfig\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.118230 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-sys-fs\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.118230 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-run-netns\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.118511 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-var-lib-cni-multus\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.118511 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-slash\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.118610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118516 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0e098fe-1a43-4895-a7e0-ab36688d5b77-host\") pod \"node-ca-k58bv\" (UID: \"c0e098fe-1a43-4895-a7e0-ab36688d5b77\") " pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.118610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118409 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-var-lib-cni-multus\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.118610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118358 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-run-netns\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.118610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-lib-modules\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.118790 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-socket-dir-parent\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.118790 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/942c56c3-9856-4d26-82a4-f9d7c1adc1f9-host-slash\") pod \"iptables-alerter-fk57z\" (UID: \"942c56c3-9856-4d26-82a4-f9d7c1adc1f9\") " pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.118790 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a715e5d9-2205-4873-b97a-6e9a53d9acca-env-overrides\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.118790 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118724 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-socket-dir-parent\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.118790 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118746 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-socket-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.118790 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f84094c-d325-462d-8f1b-a71c48ad5524-cni-binary-copy\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.119040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118775 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/942c56c3-9856-4d26-82a4-f9d7c1adc1f9-host-slash\") pod \"iptables-alerter-fk57z\" (UID: \"942c56c3-9856-4d26-82a4-f9d7c1adc1f9\") " pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.119040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.118819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-system-cni-dir\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.119296 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-os-release\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.119296 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119276 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-system-cni-dir\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.119535 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119348 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-os-release\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.119535 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8xv\" (UniqueName: \"kubernetes.io/projected/942c56c3-9856-4d26-82a4-f9d7c1adc1f9-kube-api-access-zg8xv\") pod \"iptables-alerter-fk57z\" (UID: \"942c56c3-9856-4d26-82a4-f9d7c1adc1f9\") " pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.119535 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-etc-openvswitch\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.119535 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119412 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-modprobe-d\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.119535 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-sysctl-d\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.119535 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-systemd\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.119535 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-sysctl-conf\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.119535 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119519 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-host\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.119535 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt796\" (UniqueName: \"kubernetes.io/projected/f085aed6-890e-4ac3-9129-6fc81f1f3d44-kube-api-access-tt796\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-device-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0a18e54-07ff-4255-b8b9-998883464104-kubelet-config\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0a18e54-07ff-4255-b8b9-998883464104-dbus\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-tuning-conf-dir\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-conf-dir\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-run-openvswitch\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119740 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-conf-dir\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119743 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-cni-bin\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119738 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f84094c-d325-462d-8f1b-a71c48ad5524-cni-binary-copy\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-os-release\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-var-lib-cni-bin\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119858 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-var-lib-cni-bin\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119862 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-os-release\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119862 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-tuning-conf-dir\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9q9w\" (UniqueName: \"kubernetes.io/projected/69a4d19c-0866-442f-af12-6bd23fc4c17e-kube-api-access-r9q9w\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.119908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-kubelet\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.119943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e66429f-dbc0-452f-9fe8-795b63558fd3-agent-certs\") pod \"konnectivity-agent-r28zn\" (UID: \"1e66429f-dbc0-452f-9fe8-795b63558fd3\") " pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120056 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0e098fe-1a43-4895-a7e0-ab36688d5b77-serviceca\") pod \"node-ca-k58bv\" (UID: \"c0e098fe-1a43-4895-a7e0-ab36688d5b77\") " pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-system-cni-dir\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-cni-dir\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-cnibin\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120137 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-system-cni-dir\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120191 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-cni-dir\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120208 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-cnibin\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/942c56c3-9856-4d26-82a4-f9d7c1adc1f9-iptables-alerter-script\") pod \"iptables-alerter-fk57z\" (UID: \"942c56c3-9856-4d26-82a4-f9d7c1adc1f9\") " pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-cnibin\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-var-lib-kubelet\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f748b10a-2034-4330-9292-f4781f1b4359-tmp-dir\") pod \"node-resolver-5fnqb\" (UID: \"f748b10a-2034-4330-9292-f4781f1b4359\") " pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-kubernetes\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120338 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f84094c-d325-462d-8f1b-a71c48ad5524-cnibin\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.120695 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-tuned\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a715e5d9-2205-4873-b97a-6e9a53d9acca-ovnkube-config\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120397 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-var-lib-kubelet\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f748b10a-2034-4330-9292-f4781f1b4359-hosts-file\") pod \"node-resolver-5fnqb\" (UID: \"f748b10a-2034-4330-9292-f4781f1b4359\") " pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7plss\" (UniqueName: \"kubernetes.io/projected/9f84094c-d325-462d-8f1b-a71c48ad5524-kube-api-access-7plss\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-run-systemd\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-node-log\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-log-socket\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-cni-netd\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-run\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f84094c-d325-462d-8f1b-a71c48ad5524-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-systemd-units\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-var-lib-openvswitch\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120660 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f748b10a-2034-4330-9292-f4781f1b4359-tmp-dir\") pod \"node-resolver-5fnqb\" (UID: \"f748b10a-2034-4330-9292-f4781f1b4359\") " pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a715e5d9-2205-4873-b97a-6e9a53d9acca-ovnkube-script-lib\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-var-lib-kubelet\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.121497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f085aed6-890e-4ac3-9129-6fc81f1f3d44-tmp\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75jz8\" (UniqueName: \"kubernetes.io/projected/d3aec8fe-cd66-48f3-970b-0e65561d692b-kube-api-access-75jz8\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120754 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f748b10a-2034-4330-9292-f4781f1b4359-hosts-file\") pod \"node-resolver-5fnqb\" (UID: \"f748b10a-2034-4330-9292-f4781f1b4359\") " pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-daemon-config\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69a4d19c-0866-442f-af12-6bd23fc4c17e-cni-binary-copy\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snrj\" (UniqueName: \"kubernetes.io/projected/86b6d922-1a55-4844-b308-127fc062e056-kube-api-access-8snrj\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-run-ovn\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.120932 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e66429f-dbc0-452f-9fe8-795b63558fd3-konnectivity-ca\") pod \"konnectivity-agent-r28zn\" (UID: \"1e66429f-dbc0-452f-9fe8-795b63558fd3\") " pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.121028 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-run-k8s-cni-cncf-io\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.121087 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs podName:86b6d922-1a55-4844-b308-127fc062e056 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:37.621068034 +0000 UTC m=+2.975814335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs") pod "network-metrics-daemon-wqxvp" (UID: "86b6d922-1a55-4844-b308-127fc062e056") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121110 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-run-k8s-cni-cncf-io\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-run-multus-certs\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-etc-kubernetes\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a715e5d9-2205-4873-b97a-6e9a53d9acca-ovn-node-metrics-cert\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt5zq\" (UniqueName: \"kubernetes.io/projected/a715e5d9-2205-4873-b97a-6e9a53d9acca-kube-api-access-qt5zq\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121265 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-etc-kubernetes\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121267 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-host-run-multus-certs\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgqv\" (UniqueName: \"kubernetes.io/projected/c0e098fe-1a43-4895-a7e0-ab36688d5b77-kube-api-access-6tgqv\") pod \"node-ca-k58bv\" (UID: \"c0e098fe-1a43-4895-a7e0-ab36688d5b77\") " pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-sys\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121368 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f84094c-d325-462d-8f1b-a71c48ad5524-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-registration-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpr9x\" (UniqueName: \"kubernetes.io/projected/f748b10a-2034-4330-9292-f4781f1b4359-kube-api-access-qpr9x\") pod \"node-resolver-5fnqb\" (UID: \"f748b10a-2034-4330-9292-f4781f1b4359\") " pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121431 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/942c56c3-9856-4d26-82a4-f9d7c1adc1f9-iptables-alerter-script\") pod \"iptables-alerter-fk57z\" (UID: \"942c56c3-9856-4d26-82a4-f9d7c1adc1f9\") " pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f84094c-d325-462d-8f1b-a71c48ad5524-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-hostroot\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-run-netns\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121533 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121535 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69a4d19c-0866-442f-af12-6bd23fc4c17e-cni-binary-copy\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121555 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69a4d19c-0866-442f-af12-6bd23fc4c17e-hostroot\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.122609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.121554 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69a4d19c-0866-442f-af12-6bd23fc4c17e-multus-daemon-config\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.123185 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.122749 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f84094c-d325-462d-8f1b-a71c48ad5524-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.129813 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.129792 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jun 11 20:25:37.132996 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.132957 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snrj\" (UniqueName: \"kubernetes.io/projected/86b6d922-1a55-4844-b308-127fc062e056-kube-api-access-8snrj\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:37.134166 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.134133 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 11 20:25:37.134166 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.134160 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 11 20:25:37.134327 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.134174 2564 projected.go:196] Error preparing data for projected volume kube-api-access-pckq7 for pod openshift-network-diagnostics/network-check-target-jv82t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:37.134327 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.134249 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7 podName:24fe286b-e2a1-4004-87d8-450503f21c90 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:37.634231645 +0000 UTC m=+2.988977956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pckq7" (UniqueName: "kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7") pod "network-check-target-jv82t" (UID: "24fe286b-e2a1-4004-87d8-450503f21c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:37.135934 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.135905 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8xv\" (UniqueName: \"kubernetes.io/projected/942c56c3-9856-4d26-82a4-f9d7c1adc1f9-kube-api-access-zg8xv\") pod \"iptables-alerter-fk57z\" (UID: \"942c56c3-9856-4d26-82a4-f9d7c1adc1f9\") " pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.136381 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.136266 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7plss\" (UniqueName: \"kubernetes.io/projected/9f84094c-d325-462d-8f1b-a71c48ad5524-kube-api-access-7plss\") pod \"multus-additional-cni-plugins-97sxc\" (UID: \"9f84094c-d325-462d-8f1b-a71c48ad5524\") " pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.136771 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.136745 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpr9x\" (UniqueName: \"kubernetes.io/projected/f748b10a-2034-4330-9292-f4781f1b4359-kube-api-access-qpr9x\") pod \"node-resolver-5fnqb\" (UID: \"f748b10a-2034-4330-9292-f4781f1b4359\") " pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.137145 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.137126 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9q9w\" (UniqueName: \"kubernetes.io/projected/69a4d19c-0866-442f-af12-6bd23fc4c17e-kube-api-access-r9q9w\") pod \"multus-v8xfv\" (UID: \"69a4d19c-0866-442f-af12-6bd23fc4c17e\") " pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.221863 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.221827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-tuned\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.221863 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.221867 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a715e5d9-2205-4873-b97a-6e9a53d9acca-ovnkube-config\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.221896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-run-systemd\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.221911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-node-log\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.221927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-log-socket\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.221948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-cni-netd\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222019 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-cni-netd\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222034 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-run-systemd\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222077 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-node-log\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-run\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.222101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222104 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-log-socket\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-systemd-units\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-var-lib-openvswitch\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a715e5d9-2205-4873-b97a-6e9a53d9acca-ovnkube-script-lib\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-var-lib-kubelet\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f085aed6-890e-4ac3-9129-6fc81f1f3d44-tmp\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75jz8\" (UniqueName: \"kubernetes.io/projected/d3aec8fe-cd66-48f3-970b-0e65561d692b-kube-api-access-75jz8\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-run-ovn\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222310 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-var-lib-openvswitch\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222346 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-run\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222361 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-systemd-units\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222365 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e66429f-dbc0-452f-9fe8-795b63558fd3-konnectivity-ca\") pod \"konnectivity-agent-r28zn\" (UID: \"1e66429f-dbc0-452f-9fe8-795b63558fd3\") " pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222404 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-run-ovn\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a715e5d9-2205-4873-b97a-6e9a53d9acca-ovn-node-metrics-cert\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.222483 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt5zq\" (UniqueName: \"kubernetes.io/projected/a715e5d9-2205-4873-b97a-6e9a53d9acca-kube-api-access-qt5zq\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgqv\" (UniqueName: \"kubernetes.io/projected/c0e098fe-1a43-4895-a7e0-ab36688d5b77-kube-api-access-6tgqv\") pod \"node-ca-k58bv\" (UID: \"c0e098fe-1a43-4895-a7e0-ab36688d5b77\") " pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-sys\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-registration-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-run-netns\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222606 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-sysconfig\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222681 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a715e5d9-2205-4873-b97a-6e9a53d9acca-ovnkube-config\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-sys-fs\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-slash\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0e098fe-1a43-4895-a7e0-ab36688d5b77-host\") pod \"node-ca-k58bv\" (UID: \"c0e098fe-1a43-4895-a7e0-ab36688d5b77\") " pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-lib-modules\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a715e5d9-2205-4873-b97a-6e9a53d9acca-env-overrides\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-socket-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-etc-openvswitch\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-modprobe-d\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.223294 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222930 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e66429f-dbc0-452f-9fe8-795b63558fd3-konnectivity-ca\") pod \"konnectivity-agent-r28zn\" (UID: \"1e66429f-dbc0-452f-9fe8-795b63558fd3\") " pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-sysctl-d\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.222981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-systemd\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-sysctl-conf\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223029 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-var-lib-kubelet\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-host\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt796\" (UniqueName: \"kubernetes.io/projected/f085aed6-890e-4ac3-9129-6fc81f1f3d44-kube-api-access-tt796\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223094 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-sys-fs\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223148 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a715e5d9-2205-4873-b97a-6e9a53d9acca-ovnkube-script-lib\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223213 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-systemd\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223226 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-socket-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-device-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223310 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0a18e54-07ff-4255-b8b9-998883464104-kubelet-config\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0a18e54-07ff-4255-b8b9-998883464104-dbus\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-run-openvswitch\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-cni-bin\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.224098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223462 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a715e5d9-2205-4873-b97a-6e9a53d9acca-env-overrides\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223500 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-slash\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223523 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-registration-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223626 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-etc-openvswitch\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223632 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-sysctl-conf\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223692 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0e098fe-1a43-4895-a7e0-ab36688d5b77-host\") pod \"node-ca-k58bv\" (UID: \"c0e098fe-1a43-4895-a7e0-ab36688d5b77\") " pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223742 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-modprobe-d\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223791 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-lib-modules\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.223882 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223907 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-cni-bin\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223916 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-sysctl-d\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.223975 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret podName:b0a18e54-07ff-4255-b8b9-998883464104 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:37.723941862 +0000 UTC m=+3.078688155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret") pod "global-pull-secret-syncer-v8w2z" (UID: "b0a18e54-07ff-4255-b8b9-998883464104") : object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223978 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-device-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.223990 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-host\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224024 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224063 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224064 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-kubelet\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.224862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224093 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-run-netns\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-host-kubelet\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224117 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0a18e54-07ff-4255-b8b9-998883464104-kubelet-config\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224170 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-sys\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e66429f-dbc0-452f-9fe8-795b63558fd3-agent-certs\") pod \"konnectivity-agent-r28zn\" (UID: \"1e66429f-dbc0-452f-9fe8-795b63558fd3\") " pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224231 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-sysconfig\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0e098fe-1a43-4895-a7e0-ab36688d5b77-serviceca\") pod \"node-ca-k58bv\" (UID: \"c0e098fe-1a43-4895-a7e0-ab36688d5b77\") " pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224247 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a715e5d9-2205-4873-b97a-6e9a53d9acca-run-openvswitch\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224295 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3aec8fe-cd66-48f3-970b-0e65561d692b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224336 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0a18e54-07ff-4255-b8b9-998883464104-dbus\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-kubernetes\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224539 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-kubernetes\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.224736 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0e098fe-1a43-4895-a7e0-ab36688d5b77-serviceca\") pod \"node-ca-k58bv\" (UID: \"c0e098fe-1a43-4895-a7e0-ab36688d5b77\") " pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.225702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.225002 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f085aed6-890e-4ac3-9129-6fc81f1f3d44-etc-tuned\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.226375 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.226207 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f085aed6-890e-4ac3-9129-6fc81f1f3d44-tmp\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.226710 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.226685 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a715e5d9-2205-4873-b97a-6e9a53d9acca-ovn-node-metrics-cert\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.226834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.226812 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e66429f-dbc0-452f-9fe8-795b63558fd3-agent-certs\") pod \"konnectivity-agent-r28zn\" (UID: \"1e66429f-dbc0-452f-9fe8-795b63558fd3\") " pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:37.233519 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.233494 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt796\" (UniqueName: \"kubernetes.io/projected/f085aed6-890e-4ac3-9129-6fc81f1f3d44-kube-api-access-tt796\") pod \"tuned-qw9mb\" (UID: \"f085aed6-890e-4ac3-9129-6fc81f1f3d44\") " pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.234103 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.234081 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt5zq\" (UniqueName: \"kubernetes.io/projected/a715e5d9-2205-4873-b97a-6e9a53d9acca-kube-api-access-qt5zq\") pod \"ovnkube-node-x5z7f\" (UID: \"a715e5d9-2205-4873-b97a-6e9a53d9acca\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.234520 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.234500 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgqv\" (UniqueName: \"kubernetes.io/projected/c0e098fe-1a43-4895-a7e0-ab36688d5b77-kube-api-access-6tgqv\") pod \"node-ca-k58bv\" (UID: \"c0e098fe-1a43-4895-a7e0-ab36688d5b77\") " pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.234610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.234545 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75jz8\" (UniqueName: \"kubernetes.io/projected/d3aec8fe-cd66-48f3-970b-0e65561d692b-kube-api-access-75jz8\") pod \"aws-ebs-csi-driver-node-wmwk6\" (UID: \"d3aec8fe-cd66-48f3-970b-0e65561d692b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.279918 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.279838 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 11 20:25:37.313258 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.313223 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fk57z" Jun 11 20:25:37.322191 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.322168 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5fnqb" Jun 11 20:25:37.328910 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.328886 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-97sxc" Jun 11 20:25:37.335495 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.335473 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v8xfv" Jun 11 20:25:37.342096 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.342074 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" Jun 11 20:25:37.350750 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.350725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:25:37.355407 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.355389 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:37.362793 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.362769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" Jun 11 20:25:37.369229 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.369205 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k58bv" Jun 11 20:25:37.627248 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.627167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:37.627395 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.627335 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:37.627445 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.627398 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs podName:86b6d922-1a55-4844-b308-127fc062e056 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:38.627383903 +0000 UTC m=+3.982130196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs") pod "network-metrics-daemon-wqxvp" (UID: "86b6d922-1a55-4844-b308-127fc062e056") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:37.728542 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.728503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:37.728726 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:37.728578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:37.728726 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.728701 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:37.728817 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.728752 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 11 20:25:37.728817 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.728764 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret podName:b0a18e54-07ff-4255-b8b9-998883464104 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:38.728750092 +0000 UTC m=+4.083496387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret") pod "global-pull-secret-syncer-v8w2z" (UID: "b0a18e54-07ff-4255-b8b9-998883464104") : object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:37.728817 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.728768 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 11 20:25:37.728817 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.728779 2564 projected.go:196] Error preparing data for projected volume kube-api-access-pckq7 for pod openshift-network-diagnostics/network-check-target-jv82t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:37.728986 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:37.728820 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7 podName:24fe286b-e2a1-4004-87d8-450503f21c90 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:38.728807795 +0000 UTC m=+4.083554092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pckq7" (UniqueName: "kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7") pod "network-check-target-jv82t" (UID: "24fe286b-e2a1-4004-87d8-450503f21c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:38.024297 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:38.024264 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf748b10a_2034_4330_9292_f4781f1b4359.slice/crio-5a416c09a668e12f2186b502eb13d9fcb4a56414c1ccfa1703c05113e1ebbcc3 WatchSource:0}: Error finding container 5a416c09a668e12f2186b502eb13d9fcb4a56414c1ccfa1703c05113e1ebbcc3: Status 404 returned error can't find the container with id 5a416c09a668e12f2186b502eb13d9fcb4a56414c1ccfa1703c05113e1ebbcc3 Jun 11 20:25:38.028697 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:38.028671 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3aec8fe_cd66_48f3_970b_0e65561d692b.slice/crio-89daf222c5fe19594ef9c6f7ad0e28d1cd9d794e36578802df3a25e9a352cec1 WatchSource:0}: Error finding container 89daf222c5fe19594ef9c6f7ad0e28d1cd9d794e36578802df3a25e9a352cec1: Status 404 returned error can't find the container with id 89daf222c5fe19594ef9c6f7ad0e28d1cd9d794e36578802df3a25e9a352cec1 Jun 11 20:25:38.041126 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:25:38.041102 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf085aed6_890e_4ac3_9129_6fc81f1f3d44.slice/crio-893a6cb65abc54e3d474d692ea376702e9bbecc574c2783962aa3d324c44cbf0 WatchSource:0}: Error finding container 893a6cb65abc54e3d474d692ea376702e9bbecc574c2783962aa3d324c44cbf0: Status 404 returned error can't find the container with id 893a6cb65abc54e3d474d692ea376702e9bbecc574c2783962aa3d324c44cbf0 Jun 11 20:25:38.065852 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.065631 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-10 20:20:37 +0000 UTC" deadline="2027-11-16 06:16:52.688423872 +0000 UTC" Jun 11 20:25:38.065993 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.065854 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12537h51m14.622575726s" Jun 11 20:25:38.070940 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.070915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:38.071067 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.070914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:38.071130 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.071074 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:38.071233 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.071208 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:38.083589 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.083501 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" event={"ID":"f085aed6-890e-4ac3-9129-6fc81f1f3d44","Type":"ContainerStarted","Data":"893a6cb65abc54e3d474d692ea376702e9bbecc574c2783962aa3d324c44cbf0"} Jun 11 20:25:38.085994 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.085944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k58bv" event={"ID":"c0e098fe-1a43-4895-a7e0-ab36688d5b77","Type":"ContainerStarted","Data":"b458ca2fc5bed117a3948b468e7c815d5c62c3880950d5a44add46feccd128c0"} Jun 11 20:25:38.087012 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.086987 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97sxc" event={"ID":"9f84094c-d325-462d-8f1b-a71c48ad5524","Type":"ContainerStarted","Data":"27228098a2163a24d35c3a0b107dde8d6cb752e62220a67736be7a7685bf11d2"} Jun 11 20:25:38.088088 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.088067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerStarted","Data":"2701fd0be011984ff7d03058d7596f60eacdd27d4a669d15cadd8e38fb3f3122"} Jun 11 20:25:38.089113 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.089091 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fk57z" event={"ID":"942c56c3-9856-4d26-82a4-f9d7c1adc1f9","Type":"ContainerStarted","Data":"3afd0f08b89113bc59ba1835d8b6e98fcca40ccb672689bf6054ae93bcd8c476"} Jun 11 20:25:38.090192 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.090166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5fnqb" event={"ID":"f748b10a-2034-4330-9292-f4781f1b4359","Type":"ContainerStarted","Data":"5a416c09a668e12f2186b502eb13d9fcb4a56414c1ccfa1703c05113e1ebbcc3"} Jun 11 20:25:38.091264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.091239 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v8xfv" event={"ID":"69a4d19c-0866-442f-af12-6bd23fc4c17e","Type":"ContainerStarted","Data":"c9a0b46675c06b6aa299ab03309d0a1fe5d141fab33bb7ca5ff0762e03c556bb"} Jun 11 20:25:38.092329 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.092308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r28zn" event={"ID":"1e66429f-dbc0-452f-9fe8-795b63558fd3","Type":"ContainerStarted","Data":"233ffb131f976cda7ec5b78328b993c31fc26fbd37adeba356742d84b5dfac40"} Jun 11 20:25:38.093145 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.093124 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" event={"ID":"d3aec8fe-cd66-48f3-970b-0e65561d692b","Type":"ContainerStarted","Data":"89daf222c5fe19594ef9c6f7ad0e28d1cd9d794e36578802df3a25e9a352cec1"} Jun 11 20:25:38.636608 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.636500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:38.636772 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.636742 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:38.636834 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.636807 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs podName:86b6d922-1a55-4844-b308-127fc062e056 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:40.63678811 +0000 UTC m=+5.991534424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs") pod "network-metrics-daemon-wqxvp" (UID: "86b6d922-1a55-4844-b308-127fc062e056") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:38.738297 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.737420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:38.738297 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:38.737505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:38.738297 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.737635 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:38.738297 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.737704 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret podName:b0a18e54-07ff-4255-b8b9-998883464104 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:40.73768221 +0000 UTC m=+6.092428518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret") pod "global-pull-secret-syncer-v8w2z" (UID: "b0a18e54-07ff-4255-b8b9-998883464104") : object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:38.738297 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.738174 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 11 20:25:38.738297 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.738197 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 11 20:25:38.738297 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.738209 2564 projected.go:196] Error preparing data for projected volume kube-api-access-pckq7 for pod openshift-network-diagnostics/network-check-target-jv82t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:38.738297 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:38.738255 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7 podName:24fe286b-e2a1-4004-87d8-450503f21c90 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:40.738239704 +0000 UTC m=+6.092986013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pckq7" (UniqueName: "kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7") pod "network-check-target-jv82t" (UID: "24fe286b-e2a1-4004-87d8-450503f21c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:39.066884 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:39.066844 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-10 20:20:37 +0000 UTC" deadline="2028-01-11 10:10:20.220797615 +0000 UTC" Jun 11 20:25:39.066884 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:39.066881 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13885h44m41.153920783s" Jun 11 20:25:39.074855 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:39.074830 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:39.075012 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:39.074976 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:39.112209 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:39.112175 2564 generic.go:356] "Generic (PLEG): container finished" podID="a24d161ecf306ac01e33da9ffb33262b" containerID="60c1ad00607a8bc0f43a3c75c42829275cb1f009a9d8fe5c6a21577437679bff" exitCode=0 Jun 11 20:25:39.112371 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:39.112283 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" event={"ID":"a24d161ecf306ac01e33da9ffb33262b","Type":"ContainerDied","Data":"60c1ad00607a8bc0f43a3c75c42829275cb1f009a9d8fe5c6a21577437679bff"} Jun 11 20:25:39.126582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:39.126120 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal" event={"ID":"99904d988681166720269c6ecc3484aa","Type":"ContainerStarted","Data":"227d2da056bf2345f199b338252d5bf21783b16ba71af21be79002a2756d221a"} Jun 11 20:25:40.070118 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:40.070083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:40.070634 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.070216 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:40.070634 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:40.070291 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:40.070634 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.070356 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:40.144180 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:40.144034 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" event={"ID":"a24d161ecf306ac01e33da9ffb33262b","Type":"ContainerStarted","Data":"aabaacfba9b559abf806e41291a3a49a6252d28f9e4951ce9ab619e87b6913b6"} Jun 11 20:25:40.158835 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:40.158763 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-254.ec2.internal" podStartSLOduration=4.158743168 podStartE2EDuration="4.158743168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:25:39.144400991 +0000 UTC m=+4.499147306" watchObservedRunningTime="2026-06-11 20:25:40.158743168 +0000 UTC m=+5.513489485" Jun 11 20:25:40.159596 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:40.159139 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-254.ec2.internal" podStartSLOduration=4.159126861 podStartE2EDuration="4.159126861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:25:40.158114742 +0000 UTC m=+5.512861060" watchObservedRunningTime="2026-06-11 20:25:40.159126861 +0000 UTC m=+5.513873178" Jun 11 20:25:40.654926 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:40.654752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:40.655141 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.654978 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:40.655141 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.655040 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs podName:86b6d922-1a55-4844-b308-127fc062e056 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:44.655023994 +0000 UTC m=+10.009770287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs") pod "network-metrics-daemon-wqxvp" (UID: "86b6d922-1a55-4844-b308-127fc062e056") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:40.755414 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:40.755377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:40.755610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:40.755451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:40.755610 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.755594 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 11 20:25:40.755713 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.755614 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 11 20:25:40.755713 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.755627 2564 projected.go:196] Error preparing data for projected volume kube-api-access-pckq7 for pod openshift-network-diagnostics/network-check-target-jv82t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:40.755713 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.755684 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7 podName:24fe286b-e2a1-4004-87d8-450503f21c90 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:44.755666816 +0000 UTC m=+10.110413127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pckq7" (UniqueName: "kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7") pod "network-check-target-jv82t" (UID: "24fe286b-e2a1-4004-87d8-450503f21c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:40.756112 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.756093 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:40.756198 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:40.756150 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret podName:b0a18e54-07ff-4255-b8b9-998883464104 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:44.756135146 +0000 UTC m=+10.110881456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret") pod "global-pull-secret-syncer-v8w2z" (UID: "b0a18e54-07ff-4255-b8b9-998883464104") : object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:41.074926 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:41.074847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:41.075370 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:41.075003 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:42.071214 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:42.070578 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:42.071214 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:42.070721 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:42.071214 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:42.070785 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:42.071214 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:42.070857 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:43.070622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:43.070593 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:43.071015 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:43.070726 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:44.070729 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:44.070697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:44.071217 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.070831 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:44.071217 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:44.070914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:44.071217 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.071040 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:44.682190 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:44.682123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:44.682373 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.682247 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:44.682373 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.682324 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs podName:86b6d922-1a55-4844-b308-127fc062e056 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:52.682304348 +0000 UTC m=+18.037050641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs") pod "network-metrics-daemon-wqxvp" (UID: "86b6d922-1a55-4844-b308-127fc062e056") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:44.782803 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:44.782771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:44.783024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:44.782844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:44.783024 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.782944 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:44.783024 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.782973 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 11 20:25:44.783024 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.782996 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 11 20:25:44.783024 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.783009 2564 projected.go:196] Error preparing data for projected volume kube-api-access-pckq7 for pod openshift-network-diagnostics/network-check-target-jv82t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:44.783024 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.783025 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret podName:b0a18e54-07ff-4255-b8b9-998883464104 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:52.783007341 +0000 UTC m=+18.137753652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret") pod "global-pull-secret-syncer-v8w2z" (UID: "b0a18e54-07ff-4255-b8b9-998883464104") : object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:44.783348 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:44.783061 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7 podName:24fe286b-e2a1-4004-87d8-450503f21c90 nodeName:}" failed. No retries permitted until 2026-06-11 20:25:52.783047556 +0000 UTC m=+18.137793851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pckq7" (UniqueName: "kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7") pod "network-check-target-jv82t" (UID: "24fe286b-e2a1-4004-87d8-450503f21c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:45.070924 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:45.070894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:45.071372 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:45.071036 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:46.069943 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:46.069889 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:46.070135 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:46.070045 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:46.070135 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:46.070098 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:46.070263 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:46.070216 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:47.070452 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:47.070324 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:47.070898 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:47.070483 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:48.070325 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.070169 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:48.070422 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:48.070403 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:48.070468 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.070221 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:48.070573 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:48.070551 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:48.162712 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.162671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" event={"ID":"f085aed6-890e-4ac3-9129-6fc81f1f3d44","Type":"ContainerStarted","Data":"e3255ee64a6c120de2fb9c2fcce84e2ed18493718fe9621e675e55697c82aa82"} Jun 11 20:25:48.164423 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.164393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k58bv" event={"ID":"c0e098fe-1a43-4895-a7e0-ab36688d5b77","Type":"ContainerStarted","Data":"1417a8b9b5d0fe49b5d0abf21ab3aa94945467e70d5c2392726b742de4911ba8"} Jun 11 20:25:48.166069 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.166040 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f84094c-d325-462d-8f1b-a71c48ad5524" containerID="d149983e7b91bb74e014b3abce68e4d518ed446d7e01198db91ba3d251249e30" exitCode=0 Jun 11 20:25:48.166193 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.166120 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97sxc" event={"ID":"9f84094c-d325-462d-8f1b-a71c48ad5524","Type":"ContainerDied","Data":"d149983e7b91bb74e014b3abce68e4d518ed446d7e01198db91ba3d251249e30"} Jun 11 20:25:48.167799 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.167769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5fnqb" event={"ID":"f748b10a-2034-4330-9292-f4781f1b4359","Type":"ContainerStarted","Data":"686755c8f175a85f70d5de9dabb73850dafc1bac92471a0916d11924a3e30f8b"} Jun 11 20:25:48.169276 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.169238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r28zn" event={"ID":"1e66429f-dbc0-452f-9fe8-795b63558fd3","Type":"ContainerStarted","Data":"1b6fa482ed7b4f6b99d7f113664e1ed3e47b26288e016b9a4fb156b5e9bf78ea"} Jun 11 20:25:48.170567 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.170539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" event={"ID":"d3aec8fe-cd66-48f3-970b-0e65561d692b","Type":"ContainerStarted","Data":"c5e200ad238ae44c465eadf71766b465634901052a81218a611d960c6a1d52eb"} Jun 11 20:25:48.179716 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.179668 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qw9mb" podStartSLOduration=3.412676989 podStartE2EDuration="13.179654585s" totalImagesPullingTime="9.766977596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:25:48.179048771 +0000 UTC m=+13.533795087" watchObservedRunningTime="2026-06-11 20:25:48.179654585 +0000 UTC m=+13.534400900" Jun 11 20:25:48.191781 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.191734 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5fnqb" podStartSLOduration=3.428976806 podStartE2EDuration="13.191715538s" totalImagesPullingTime="9.762738732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:25:48.191505871 +0000 UTC m=+13.546252188" watchObservedRunningTime="2026-06-11 20:25:48.191715538 +0000 UTC m=+13.546461855" Jun 11 20:25:48.234312 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.234249 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r28zn" podStartSLOduration=3.514674628 podStartE2EDuration="13.234235819s" totalImagesPullingTime="9.719561191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:25:48.234144237 +0000 UTC m=+13.588890554" watchObservedRunningTime="2026-06-11 20:25:48.234235819 +0000 UTC m=+13.588982137" Jun 11 20:25:48.248705 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:48.248650 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k58bv" podStartSLOduration=3.5286286369999997 podStartE2EDuration="13.248631855s" totalImagesPullingTime="9.720003218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:25:48.248145015 +0000 UTC m=+13.602891344" watchObservedRunningTime="2026-06-11 20:25:48.248631855 +0000 UTC m=+13.603378173" Jun 11 20:25:49.070809 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:49.070724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:49.071393 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:49.070927 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:49.203155 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:49.203114 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:49.203859 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:49.203808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:50.070194 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:50.070162 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:50.070379 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:50.070162 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:50.070379 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:50.070290 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:50.070379 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:50.070357 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:50.176175 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:50.176136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fk57z" event={"ID":"942c56c3-9856-4d26-82a4-f9d7c1adc1f9","Type":"ContainerStarted","Data":"4df9db8f6c48b7413c048dcc4c235509a70a244a2bf778d21e7c55f2c17e9e29"} Jun 11 20:25:50.176663 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:50.176609 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:50.177428 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:50.177405 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r28zn" Jun 11 20:25:50.191024 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:50.190942 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fk57z" podStartSLOduration=5.463537335 podStartE2EDuration="15.190925793s" totalImagesPullingTime="9.727388458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:25:50.190097164 +0000 UTC m=+15.544843783" watchObservedRunningTime="2026-06-11 20:25:50.190925793 +0000 UTC m=+15.545672395" Jun 11 20:25:51.070790 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:51.070589 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:51.070790 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:51.070733 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:52.070430 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:52.070395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:52.071013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:52.070395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:52.071013 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.070528 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:52.071013 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.070638 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:52.752026 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:52.751986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:52.752211 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.752141 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:52.752211 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.752204 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs podName:86b6d922-1a55-4844-b308-127fc062e056 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:08.752190066 +0000 UTC m=+34.106936359 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs") pod "network-metrics-daemon-wqxvp" (UID: "86b6d922-1a55-4844-b308-127fc062e056") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:25:52.852726 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:52.852692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:52.852900 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:52.852749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:52.852900 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.852847 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:52.852900 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.852864 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 11 20:25:52.852900 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.852884 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 11 20:25:52.852900 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.852896 2564 projected.go:196] Error preparing data for projected volume kube-api-access-pckq7 for pod openshift-network-diagnostics/network-check-target-jv82t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:52.853100 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.852914 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret podName:b0a18e54-07ff-4255-b8b9-998883464104 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:08.852897261 +0000 UTC m=+34.207643569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret") pod "global-pull-secret-syncer-v8w2z" (UID: "b0a18e54-07ff-4255-b8b9-998883464104") : object "kube-system"/"original-pull-secret" not registered Jun 11 20:25:52.853100 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:52.852931 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7 podName:24fe286b-e2a1-4004-87d8-450503f21c90 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:08.85292152 +0000 UTC m=+34.207667829 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pckq7" (UniqueName: "kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7") pod "network-check-target-jv82t" (UID: "24fe286b-e2a1-4004-87d8-450503f21c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:25:53.070775 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:53.070691 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:53.071231 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:53.070825 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:54.070086 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:54.070055 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:54.070086 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:54.070076 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:54.070400 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:54.070223 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:54.070400 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:54.070301 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:55.071031 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:55.070994 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:55.071600 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:55.071126 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:56.070437 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:56.070257 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:56.070610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:56.070339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:56.070610 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:56.070517 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:56.070610 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:56.070598 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:57.070284 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:57.070250 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:57.070802 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:57.070396 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:58.070108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:58.070072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:25:58.070392 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:58.070072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:25:58.070392 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:58.070199 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:25:58.070392 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:58.070292 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:25:59.070221 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:59.070187 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:25:59.070381 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:25:59.070302 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:25:59.439135 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:25:59.439083 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jun 11 20:26:00.070512 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.070315 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:00.070691 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.070366 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:00.070691 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:00.070614 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:26:00.070691 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:00.070679 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:26:00.070833 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.070623 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-11T20:25:59.43911238Z","UUID":"7f830c47-e3b4-4fa7-9c1b-7494959045fa","Handler":null,"Name":"","Endpoint":""} Jun 11 20:26:00.072134 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.072112 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jun 11 20:26:00.072241 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.072149 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jun 11 20:26:00.197543 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.197500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" event={"ID":"d3aec8fe-cd66-48f3-970b-0e65561d692b","Type":"ContainerStarted","Data":"7290e97b01e42f331c31b82969dad8d8c833d746838347df8a304c08450ae8c4"} Jun 11 20:26:00.199281 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.199254 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f84094c-d325-462d-8f1b-a71c48ad5524" containerID="f549cfe013c441682cc9b999056ee023e92e620105b824543f5ea93a1ba54dd5" exitCode=0 Jun 11 20:26:00.199418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.199333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97sxc" event={"ID":"9f84094c-d325-462d-8f1b-a71c48ad5524","Type":"ContainerDied","Data":"f549cfe013c441682cc9b999056ee023e92e620105b824543f5ea93a1ba54dd5"} Jun 11 20:26:00.202928 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.202910 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:26:00.203370 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.203344 2564 generic.go:356] "Generic (PLEG): container finished" podID="a715e5d9-2205-4873-b97a-6e9a53d9acca" containerID="4fe30888ebd5fb0c97bcff64d3f6a763f132c102917de121999cf73b1f950fc0" exitCode=1 Jun 11 20:26:00.203485 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.203373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerStarted","Data":"24d4219032241c525431e2b79690255f14641f9a51f70b9810a946082bb3afc0"} Jun 11 20:26:00.203485 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.203450 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerStarted","Data":"545143d3cf8beba1943e44596d98a10836121cfa00013bb32a84d4ec576c4334"} Jun 11 20:26:00.203485 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.203483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerStarted","Data":"920f35794b1d1aee850c9e4fe42c580da22915de87ad7852a52ee0b297f846d3"} Jun 11 20:26:00.203640 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.203510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerStarted","Data":"ef23899fb65956caac6817a15d7ff6612e2ec6d4e1644410f421f0dc4255e096"} Jun 11 20:26:00.203640 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.203539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerDied","Data":"4fe30888ebd5fb0c97bcff64d3f6a763f132c102917de121999cf73b1f950fc0"} Jun 11 20:26:00.203640 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.203580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerStarted","Data":"c81233f3edb35df4549e9b9e802ed3c0f323d79dae1dd700378b8b535c0e25b6"} Jun 11 20:26:00.205020 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.204998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v8xfv" event={"ID":"69a4d19c-0866-442f-af12-6bd23fc4c17e","Type":"ContainerStarted","Data":"de607b9de7ff43d7f487e6a0824d3c369760f5e0ab9f09ed405fe24b37fee48a"} Jun 11 20:26:00.233184 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:00.233122 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v8xfv" podStartSLOduration=3.968223103 podStartE2EDuration="25.233102756s" totalImagesPullingTime="21.264879653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:26:00.232727131 +0000 UTC m=+25.587473447" watchObservedRunningTime="2026-06-11 20:26:00.233102756 +0000 UTC m=+25.587849072" Jun 11 20:26:01.070904 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:01.070862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:26:01.071593 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:01.071031 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:26:01.209429 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:01.209393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" event={"ID":"d3aec8fe-cd66-48f3-970b-0e65561d692b","Type":"ContainerStarted","Data":"8eda3c0b66f1bca3d6609d085ef9f6f7533fa8b2b91ff5af66122136b54729fc"} Jun 11 20:26:01.226082 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:01.226035 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmwk6" podStartSLOduration=3.98013201 podStartE2EDuration="26.226021248s" totalImagesPullingTime="22.245889238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:26:01.224869143 +0000 UTC m=+26.579615461" watchObservedRunningTime="2026-06-11 20:26:01.226021248 +0000 UTC m=+26.580767564" Jun 11 20:26:02.070361 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:02.070326 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:02.070511 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:02.070326 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:02.070511 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:02.070450 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:26:02.070511 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:02.070496 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:26:02.213855 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:02.213768 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f84094c-d325-462d-8f1b-a71c48ad5524" containerID="98f4e5fbde6605e45300ba0991c460757d71f1116e0b9b89afa2acaef3b2aa9e" exitCode=0 Jun 11 20:26:02.214274 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:02.213854 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97sxc" event={"ID":"9f84094c-d325-462d-8f1b-a71c48ad5524","Type":"ContainerDied","Data":"98f4e5fbde6605e45300ba0991c460757d71f1116e0b9b89afa2acaef3b2aa9e"} Jun 11 20:26:02.216958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:02.216939 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:26:02.217422 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:02.217395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerStarted","Data":"5ca17e8aeb3c9adfe963b14889fd18c8f1ecab2b760db992513aa76762ee5260"} Jun 11 20:26:03.070332 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:03.070301 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:26:03.070497 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:03.070411 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:26:03.222224 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:03.222128 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f84094c-d325-462d-8f1b-a71c48ad5524" containerID="29dda1f999fdf8a9046a62ed1746f9cf2eeecfc58ab86347e9d818dccc773951" exitCode=0 Jun 11 20:26:03.222224 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:03.222193 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97sxc" event={"ID":"9f84094c-d325-462d-8f1b-a71c48ad5524","Type":"ContainerDied","Data":"29dda1f999fdf8a9046a62ed1746f9cf2eeecfc58ab86347e9d818dccc773951"} Jun 11 20:26:04.070803 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.070762 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:04.070803 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.070801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:04.071003 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:04.070894 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:26:04.071075 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:04.071052 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:26:04.227187 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.227158 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:26:04.227650 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.227493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerStarted","Data":"43bb89b084e392cec4c83649ed6798d76c1fd21a6db3ea55ae1c77056d66a638"} Jun 11 20:26:04.227693 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.227669 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:26:04.227733 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.227718 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:26:04.227767 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.227759 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:26:04.228041 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.228023 2564 scope.go:117] "RemoveContainer" containerID="4fe30888ebd5fb0c97bcff64d3f6a763f132c102917de121999cf73b1f950fc0" Jun 11 20:26:04.244065 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.244044 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:26:04.244401 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:04.244383 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:26:05.070804 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:05.070772 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:26:05.071001 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:05.070889 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:26:05.241296 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:05.241263 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:26:05.241768 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:05.241741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" event={"ID":"a715e5d9-2205-4873-b97a-6e9a53d9acca","Type":"ContainerStarted","Data":"3ed01ea3ca8ee25a3e29864bec734e3aee4c94666fa02fc53c32e8eee26817bb"} Jun 11 20:26:05.288841 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:05.288789 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" podStartSLOduration=9.061551915 podStartE2EDuration="30.288770229s" totalImagesPullingTime="21.227218314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:26:05.288373106 +0000 UTC m=+30.643119450" watchObservedRunningTime="2026-06-11 20:26:05.288770229 +0000 UTC m=+30.643516544" Jun 11 20:26:06.070200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:06.070159 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:06.070404 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:06.070292 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:26:06.070471 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:06.070402 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:06.070548 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:06.070521 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:26:06.260947 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:06.260914 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v8w2z"] Jun 11 20:26:06.261333 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:06.261037 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:06.261333 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:06.261133 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:26:06.264538 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:06.264496 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jv82t"] Jun 11 20:26:06.264675 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:06.264594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:06.264719 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:06.264676 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:26:06.265284 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:06.265259 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wqxvp"] Jun 11 20:26:06.265376 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:06.265351 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:26:06.265466 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:06.265447 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:26:08.070496 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:08.070460 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:08.071049 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:08.070502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:08.071049 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:08.070509 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:26:08.071049 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.070587 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:26:08.071049 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.070683 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:26:08.071049 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.070732 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:26:08.764665 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:08.764628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:26:08.764807 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.764778 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:26:08.764854 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.764844 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs podName:86b6d922-1a55-4844-b308-127fc062e056 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:40.76482473 +0000 UTC m=+66.119571024 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs") pod "network-metrics-daemon-wqxvp" (UID: "86b6d922-1a55-4844-b308-127fc062e056") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 11 20:26:08.865356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:08.865326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:08.865507 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:08.865395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:08.865507 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.865497 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 11 20:26:08.865602 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.865563 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret podName:b0a18e54-07ff-4255-b8b9-998883464104 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:40.865546262 +0000 UTC m=+66.220292559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret") pod "global-pull-secret-syncer-v8w2z" (UID: "b0a18e54-07ff-4255-b8b9-998883464104") : object "kube-system"/"original-pull-secret" not registered Jun 11 20:26:08.865602 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.865502 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 11 20:26:08.865602 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.865596 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 11 20:26:08.865714 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.865609 2564 projected.go:196] Error preparing data for projected volume kube-api-access-pckq7 for pod openshift-network-diagnostics/network-check-target-jv82t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:26:08.865714 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:08.865648 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7 podName:24fe286b-e2a1-4004-87d8-450503f21c90 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:40.865637385 +0000 UTC m=+66.220383692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pckq7" (UniqueName: "kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7") pod "network-check-target-jv82t" (UID: "24fe286b-e2a1-4004-87d8-450503f21c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 11 20:26:10.070106 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:10.070072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:26:10.070467 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:10.070079 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:10.070467 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:10.070192 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:10.070467 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:10.070195 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:26:10.070467 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:10.070264 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v8w2z" podUID="b0a18e54-07ff-4255-b8b9-998883464104" Jun 11 20:26:10.070467 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:10.070334 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jv82t" podUID="24fe286b-e2a1-4004-87d8-450503f21c90" Jun 11 20:26:11.261467 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:11.261435 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f84094c-d325-462d-8f1b-a71c48ad5524" containerID="54a2e6350f15db8bd5a56cf9a39e6814733e630fee054dc197949cfdbfd4a7a8" exitCode=0 Jun 11 20:26:11.262022 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:11.261477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97sxc" event={"ID":"9f84094c-d325-462d-8f1b-a71c48ad5524","Type":"ContainerDied","Data":"54a2e6350f15db8bd5a56cf9a39e6814733e630fee054dc197949cfdbfd4a7a8"} Jun 11 20:26:12.011444 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.011414 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-254.ec2.internal" event="NodeReady" Jun 11 20:26:12.011601 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.011579 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jun 11 20:26:12.046322 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.046292 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh"] Jun 11 20:26:12.060915 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.060882 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fgbx2"] Jun 11 20:26:12.061103 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.061050 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.063229 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.063197 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jun 11 20:26:12.063509 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.063362 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jun 11 20:26:12.063509 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.063362 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jun 11 20:26:12.063509 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.063450 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jm4xk\"" Jun 11 20:26:12.068823 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.068801 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jun 11 20:26:12.074952 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.073230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:12.074952 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.073275 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qpm6t"] Jun 11 20:26:12.074952 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.073237 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fgbx2" Jun 11 20:26:12.074952 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.073377 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:26:12.074952 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.074035 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:12.075456 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.075437 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 11 20:26:12.076013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.075990 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 11 20:26:12.076123 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.076074 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xphjl\"" Jun 11 20:26:12.076199 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.076133 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9v6lj\"" Jun 11 20:26:12.076284 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.075999 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 11 20:26:12.076537 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.076476 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 11 20:26:12.076802 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.076785 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jun 11 20:26:12.076900 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.076862 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jun 11 20:26:12.077018 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.076983 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7h2m\"" Jun 11 20:26:12.077106 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.077049 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jun 11 20:26:12.087674 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.087654 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh"] Jun 11 20:26:12.087786 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.087683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fgbx2"] Jun 11 20:26:12.087786 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.087698 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qpm6t"] Jun 11 20:26:12.087786 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.087780 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.090001 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.089982 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jun 11 20:26:12.090223 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.090210 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jun 11 20:26:12.090280 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.090234 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7qg4w\"" Jun 11 20:26:12.190614 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.190614 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-registry-certificates\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e98f77d6-3aa7-4ace-8987-93b443679091-config-volume\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a491ede1-e21e-4861-8023-2a5d7d331351-ca-trust-extracted\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01568624-0240-48ae-9f43-2bc8eb597dcd-cert\") pod \"ingress-canary-fgbx2\" (UID: \"01568624-0240-48ae-9f43-2bc8eb597dcd\") " pod="openshift-ingress-canary/ingress-canary-fgbx2" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-image-registry-private-configuration\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx996\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-kube-api-access-cx996\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js86p\" (UniqueName: \"kubernetes.io/projected/01568624-0240-48ae-9f43-2bc8eb597dcd-kube-api-access-js86p\") pod \"ingress-canary-fgbx2\" (UID: \"01568624-0240-48ae-9f43-2bc8eb597dcd\") " pod="openshift-ingress-canary/ingress-canary-fgbx2" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e98f77d6-3aa7-4ace-8987-93b443679091-tmp-dir\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-trusted-ca\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-installation-pull-secrets\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.190991 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.190939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td27v\" (UniqueName: \"kubernetes.io/projected/e98f77d6-3aa7-4ace-8987-93b443679091-kube-api-access-td27v\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.191353 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.191008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-bound-sa-token\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.266589 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.266556 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f84094c-d325-462d-8f1b-a71c48ad5524" containerID="213ea99905614e26526b18bea87fed300ca3005de25365b89c26cf9720767e14" exitCode=0 Jun 11 20:26:12.267128 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.266594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97sxc" event={"ID":"9f84094c-d325-462d-8f1b-a71c48ad5524","Type":"ContainerDied","Data":"213ea99905614e26526b18bea87fed300ca3005de25365b89c26cf9720767e14"} Jun 11 20:26:12.292881 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.292842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.293085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.292903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-registry-certificates\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.293085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.292938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e98f77d6-3aa7-4ace-8987-93b443679091-config-volume\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.293085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.292977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.293085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a491ede1-e21e-4861-8023-2a5d7d331351-ca-trust-extracted\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.293085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01568624-0240-48ae-9f43-2bc8eb597dcd-cert\") pod \"ingress-canary-fgbx2\" (UID: \"01568624-0240-48ae-9f43-2bc8eb597dcd\") " pod="openshift-ingress-canary/ingress-canary-fgbx2" Jun 11 20:26:12.293085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-image-registry-private-configuration\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.293356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx996\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-kube-api-access-cx996\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.293356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js86p\" (UniqueName: \"kubernetes.io/projected/01568624-0240-48ae-9f43-2bc8eb597dcd-kube-api-access-js86p\") pod \"ingress-canary-fgbx2\" (UID: \"01568624-0240-48ae-9f43-2bc8eb597dcd\") " pod="openshift-ingress-canary/ingress-canary-fgbx2" Jun 11 20:26:12.293356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e98f77d6-3aa7-4ace-8987-93b443679091-tmp-dir\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.293356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-trusted-ca\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.293356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-installation-pull-secrets\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.293356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td27v\" (UniqueName: \"kubernetes.io/projected/e98f77d6-3aa7-4ace-8987-93b443679091-kube-api-access-td27v\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.293356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.293336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-bound-sa-token\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.293793 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.293764 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 11 20:26:12.293855 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.293796 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh: secret "image-registry-tls" not found Jun 11 20:26:12.293898 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.293866 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls podName:a491ede1-e21e-4861-8023-2a5d7d331351 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:12.793841003 +0000 UTC m=+38.148587300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls") pod "image-registry-6bbdc7f8b7-9qvxh" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351") : secret "image-registry-tls" not found Jun 11 20:26:12.295219 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.295194 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e98f77d6-3aa7-4ace-8987-93b443679091-config-volume\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.295435 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.295406 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 11 20:26:12.295646 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.295630 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls podName:e98f77d6-3aa7-4ace-8987-93b443679091 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:12.795610985 +0000 UTC m=+38.150357298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls") pod "dns-default-qpm6t" (UID: "e98f77d6-3aa7-4ace-8987-93b443679091") : secret "dns-default-metrics-tls" not found Jun 11 20:26:12.295813 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.295786 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e98f77d6-3aa7-4ace-8987-93b443679091-tmp-dir\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.295953 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.295896 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a491ede1-e21e-4861-8023-2a5d7d331351-ca-trust-extracted\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.296055 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.295946 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-registry-certificates\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.296776 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.296618 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-trusted-ca\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.299789 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.299763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-installation-pull-secrets\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.299887 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.299773 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01568624-0240-48ae-9f43-2bc8eb597dcd-cert\") pod \"ingress-canary-fgbx2\" (UID: \"01568624-0240-48ae-9f43-2bc8eb597dcd\") " pod="openshift-ingress-canary/ingress-canary-fgbx2" Jun 11 20:26:12.299887 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.299837 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-image-registry-private-configuration\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.302010 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.301988 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx996\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-kube-api-access-cx996\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.302307 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.302287 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td27v\" (UniqueName: \"kubernetes.io/projected/e98f77d6-3aa7-4ace-8987-93b443679091-kube-api-access-td27v\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.302523 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.302507 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js86p\" (UniqueName: \"kubernetes.io/projected/01568624-0240-48ae-9f43-2bc8eb597dcd-kube-api-access-js86p\") pod \"ingress-canary-fgbx2\" (UID: \"01568624-0240-48ae-9f43-2bc8eb597dcd\") " pod="openshift-ingress-canary/ingress-canary-fgbx2" Jun 11 20:26:12.312157 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.312133 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-bound-sa-token\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.403381 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.403228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fgbx2" Jun 11 20:26:12.564745 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.564715 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fgbx2"] Jun 11 20:26:12.568184 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:26:12.568156 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01568624_0240_48ae_9f43_2bc8eb597dcd.slice/crio-3f2dc2c90b24e58221fad89c814f55e2b5f9165597f73f638006839bc500cd79 WatchSource:0}: Error finding container 3f2dc2c90b24e58221fad89c814f55e2b5f9165597f73f638006839bc500cd79: Status 404 returned error can't find the container with id 3f2dc2c90b24e58221fad89c814f55e2b5f9165597f73f638006839bc500cd79 Jun 11 20:26:12.797311 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.797225 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:12.797311 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:12.797267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:12.797506 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.797373 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 11 20:26:12.797506 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.797378 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 11 20:26:12.797506 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.797393 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh: secret "image-registry-tls" not found Jun 11 20:26:12.797506 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.797423 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls podName:e98f77d6-3aa7-4ace-8987-93b443679091 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:13.797410419 +0000 UTC m=+39.152156713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls") pod "dns-default-qpm6t" (UID: "e98f77d6-3aa7-4ace-8987-93b443679091") : secret "dns-default-metrics-tls" not found Jun 11 20:26:12.797506 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:12.797436 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls podName:a491ede1-e21e-4861-8023-2a5d7d331351 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:13.79742994 +0000 UTC m=+39.152176233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls") pod "image-registry-6bbdc7f8b7-9qvxh" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351") : secret "image-registry-tls" not found Jun 11 20:26:13.270094 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:13.270060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fgbx2" event={"ID":"01568624-0240-48ae-9f43-2bc8eb597dcd","Type":"ContainerStarted","Data":"3f2dc2c90b24e58221fad89c814f55e2b5f9165597f73f638006839bc500cd79"} Jun 11 20:26:13.273118 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:13.273093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97sxc" event={"ID":"9f84094c-d325-462d-8f1b-a71c48ad5524","Type":"ContainerStarted","Data":"8efdb759c5dfa9b8cfe548f1340415a9336d99800e469d1a300fbd9cb871656b"} Jun 11 20:26:13.294680 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:13.294618 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-97sxc" podStartSLOduration=8.378197815 podStartE2EDuration="38.294596069s" totalImagesPullingTime="29.916398254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:26:13.292830979 +0000 UTC m=+38.647577335" watchObservedRunningTime="2026-06-11 20:26:13.294596069 +0000 UTC m=+38.649342386" Jun 11 20:26:13.806612 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:13.806576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:13.806778 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:13.806620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:13.806778 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:13.806750 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 11 20:26:13.806896 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:13.806817 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls podName:e98f77d6-3aa7-4ace-8987-93b443679091 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:15.806797678 +0000 UTC m=+41.161543984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls") pod "dns-default-qpm6t" (UID: "e98f77d6-3aa7-4ace-8987-93b443679091") : secret "dns-default-metrics-tls" not found Jun 11 20:26:13.806896 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:13.806750 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 11 20:26:13.806896 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:13.806859 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh: secret "image-registry-tls" not found Jun 11 20:26:13.807052 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:13.806914 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls podName:a491ede1-e21e-4861-8023-2a5d7d331351 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:15.806898623 +0000 UTC m=+41.161644936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls") pod "image-registry-6bbdc7f8b7-9qvxh" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351") : secret "image-registry-tls" not found Jun 11 20:26:15.280463 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:15.280421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fgbx2" event={"ID":"01568624-0240-48ae-9f43-2bc8eb597dcd","Type":"ContainerStarted","Data":"ff2c8f8eebc3ceb78f245cf90a7a3bf822455653227f40142d29621062fe091f"} Jun 11 20:26:15.301499 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:15.301448 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fgbx2" podStartSLOduration=0.752796615 podStartE2EDuration="3.301432856s" totalImagesPullingTime="2.548636241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:26:15.300527404 +0000 UTC m=+40.655273721" watchObservedRunningTime="2026-06-11 20:26:15.301432856 +0000 UTC m=+40.656179213" Jun 11 20:26:15.824344 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:15.824304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:15.824525 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:15.824353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:15.824525 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:15.824455 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 11 20:26:15.824525 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:15.824461 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 11 20:26:15.824525 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:15.824481 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh: secret "image-registry-tls" not found Jun 11 20:26:15.824525 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:15.824510 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls podName:e98f77d6-3aa7-4ace-8987-93b443679091 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:19.824496501 +0000 UTC m=+45.179242795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls") pod "dns-default-qpm6t" (UID: "e98f77d6-3aa7-4ace-8987-93b443679091") : secret "dns-default-metrics-tls" not found Jun 11 20:26:15.824699 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:15.824539 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls podName:a491ede1-e21e-4861-8023-2a5d7d331351 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:19.824525587 +0000 UTC m=+45.179271881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls") pod "image-registry-6bbdc7f8b7-9qvxh" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351") : secret "image-registry-tls" not found Jun 11 20:26:19.864411 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:19.864369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:19.865066 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:19.864459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:19.865066 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:19.864541 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 11 20:26:19.865066 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:19.864543 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 11 20:26:19.865066 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:19.864632 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls podName:e98f77d6-3aa7-4ace-8987-93b443679091 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:27.864610469 +0000 UTC m=+53.219356779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls") pod "dns-default-qpm6t" (UID: "e98f77d6-3aa7-4ace-8987-93b443679091") : secret "dns-default-metrics-tls" not found Jun 11 20:26:19.865066 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:19.864552 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh: secret "image-registry-tls" not found Jun 11 20:26:19.865066 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:19.864679 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls podName:a491ede1-e21e-4861-8023-2a5d7d331351 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:27.864667186 +0000 UTC m=+53.219413480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls") pod "image-registry-6bbdc7f8b7-9qvxh" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351") : secret "image-registry-tls" not found Jun 11 20:26:27.916558 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:27.916517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:27.916558 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:27.916559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:27.917274 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:27.916665 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 11 20:26:27.917274 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:27.916675 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 11 20:26:27.917274 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:27.916695 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh: secret "image-registry-tls" not found Jun 11 20:26:27.917274 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:27.916719 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls podName:e98f77d6-3aa7-4ace-8987-93b443679091 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:43.91670622 +0000 UTC m=+69.271452515 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls") pod "dns-default-qpm6t" (UID: "e98f77d6-3aa7-4ace-8987-93b443679091") : secret "dns-default-metrics-tls" not found Jun 11 20:26:27.917274 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:27.916749 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls podName:a491ede1-e21e-4861-8023-2a5d7d331351 nodeName:}" failed. No retries permitted until 2026-06-11 20:26:43.916731293 +0000 UTC m=+69.271477586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls") pod "image-registry-6bbdc7f8b7-9qvxh" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351") : secret "image-registry-tls" not found Jun 11 20:26:36.254808 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:36.254780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5z7f" Jun 11 20:26:40.806198 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:40.806157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:26:40.808706 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:40.808689 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 11 20:26:40.817412 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:40.817392 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 11 20:26:40.817498 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:40.817487 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs podName:86b6d922-1a55-4844-b308-127fc062e056 nodeName:}" failed. No retries permitted until 2026-06-11 20:27:44.817467602 +0000 UTC m=+130.172213896 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs") pod "network-metrics-daemon-wqxvp" (UID: "86b6d922-1a55-4844-b308-127fc062e056") : secret "metrics-daemon-secret" not found Jun 11 20:26:40.906703 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:40.906669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:40.906851 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:40.906737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:40.909142 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:40.909121 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 11 20:26:40.909232 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:40.909179 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 11 20:26:40.919676 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:40.919650 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 11 20:26:40.921858 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:40.921831 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0a18e54-07ff-4255-b8b9-998883464104-original-pull-secret\") pod \"global-pull-secret-syncer-v8w2z\" (UID: \"b0a18e54-07ff-4255-b8b9-998883464104\") " pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:40.931077 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:40.931050 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pckq7\" (UniqueName: \"kubernetes.io/projected/24fe286b-e2a1-4004-87d8-450503f21c90-kube-api-access-pckq7\") pod \"network-check-target-jv82t\" (UID: \"24fe286b-e2a1-4004-87d8-450503f21c90\") " pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:41.187218 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:41.187134 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xphjl\"" Jun 11 20:26:41.195259 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:41.195233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:41.198616 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:41.198595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v8w2z" Jun 11 20:26:41.354803 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:41.354703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jv82t"] Jun 11 20:26:41.357668 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:26:41.357638 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fe286b_e2a1_4004_87d8_450503f21c90.slice/crio-f135ac1e21ea9792f04829416fdf0e986e9bde6d4b58765adb7391c50e44e0e7 WatchSource:0}: Error finding container f135ac1e21ea9792f04829416fdf0e986e9bde6d4b58765adb7391c50e44e0e7: Status 404 returned error can't find the container with id f135ac1e21ea9792f04829416fdf0e986e9bde6d4b58765adb7391c50e44e0e7 Jun 11 20:26:41.370885 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:41.370858 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v8w2z"] Jun 11 20:26:41.374038 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:26:41.374013 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a18e54_07ff_4255_b8b9_998883464104.slice/crio-945177dbd6e26922e40f26b7576793c638c1f86bb4a0371efe4d9382c879a42e WatchSource:0}: Error finding container 945177dbd6e26922e40f26b7576793c638c1f86bb4a0371efe4d9382c879a42e: Status 404 returned error can't find the container with id 945177dbd6e26922e40f26b7576793c638c1f86bb4a0371efe4d9382c879a42e Jun 11 20:26:42.347978 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:42.347928 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v8w2z" event={"ID":"b0a18e54-07ff-4255-b8b9-998883464104","Type":"ContainerStarted","Data":"945177dbd6e26922e40f26b7576793c638c1f86bb4a0371efe4d9382c879a42e"} Jun 11 20:26:42.349067 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:42.349041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jv82t" event={"ID":"24fe286b-e2a1-4004-87d8-450503f21c90","Type":"ContainerStarted","Data":"f135ac1e21ea9792f04829416fdf0e986e9bde6d4b58765adb7391c50e44e0e7"} Jun 11 20:26:43.930190 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:43.930145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:26:43.930666 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:43.930200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:26:43.930666 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:43.930316 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 11 20:26:43.930666 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:43.930329 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 11 20:26:43.930666 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:43.930341 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh: secret "image-registry-tls" not found Jun 11 20:26:43.930666 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:43.930417 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls podName:e98f77d6-3aa7-4ace-8987-93b443679091 nodeName:}" failed. No retries permitted until 2026-06-11 20:27:15.930398603 +0000 UTC m=+101.285144897 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls") pod "dns-default-qpm6t" (UID: "e98f77d6-3aa7-4ace-8987-93b443679091") : secret "dns-default-metrics-tls" not found Jun 11 20:26:43.930666 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:26:43.930439 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls podName:a491ede1-e21e-4861-8023-2a5d7d331351 nodeName:}" failed. No retries permitted until 2026-06-11 20:27:15.930428992 +0000 UTC m=+101.285175288 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls") pod "image-registry-6bbdc7f8b7-9qvxh" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351") : secret "image-registry-tls" not found Jun 11 20:26:45.360207 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:45.360164 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jv82t" event={"ID":"24fe286b-e2a1-4004-87d8-450503f21c90","Type":"ContainerStarted","Data":"d9f3143521bc77e8a2f6f54354c7f066259555939bb833faa90f08df9c03f075"} Jun 11 20:26:45.360658 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:45.360320 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:26:45.376447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:45.376390 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jv82t" podStartSLOduration=67.291867527 podStartE2EDuration="1m10.376368295s" totalImagesPullingTime="3.084500768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:26:45.375205742 +0000 UTC m=+70.729952085" watchObservedRunningTime="2026-06-11 20:26:45.376368295 +0000 UTC m=+70.731114589" Jun 11 20:26:48.370180 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:48.370142 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v8w2z" event={"ID":"b0a18e54-07ff-4255-b8b9-998883464104","Type":"ContainerStarted","Data":"26923902701e2dc9a5e8d4c438cdf048595306b0587c7faa25a3d27a17365ba7"} Jun 11 20:26:48.384147 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:26:48.384093 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v8w2z" podStartSLOduration=67.247846805 podStartE2EDuration="1m13.38407649s" totalImagesPullingTime="6.136229685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:26:48.383613023 +0000 UTC m=+73.738359339" watchObservedRunningTime="2026-06-11 20:26:48.38407649 +0000 UTC m=+73.738822788" Jun 11 20:27:15.962224 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:27:15.962130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:27:15.962224 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:27:15.962195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:27:15.962631 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:27:15.962282 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 11 20:27:15.962631 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:27:15.962296 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh: secret "image-registry-tls" not found Jun 11 20:27:15.962631 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:27:15.962296 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 11 20:27:15.962631 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:27:15.962364 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls podName:a491ede1-e21e-4861-8023-2a5d7d331351 nodeName:}" failed. No retries permitted until 2026-06-11 20:28:19.96234647 +0000 UTC m=+165.317092775 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls") pod "image-registry-6bbdc7f8b7-9qvxh" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351") : secret "image-registry-tls" not found Jun 11 20:27:15.962631 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:27:15.962378 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls podName:e98f77d6-3aa7-4ace-8987-93b443679091 nodeName:}" failed. No retries permitted until 2026-06-11 20:28:19.962372374 +0000 UTC m=+165.317118668 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls") pod "dns-default-qpm6t" (UID: "e98f77d6-3aa7-4ace-8987-93b443679091") : secret "dns-default-metrics-tls" not found Jun 11 20:27:16.364699 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:27:16.364669 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jv82t" Jun 11 20:27:44.857286 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:27:44.857224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:27:44.857754 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:27:44.857405 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 11 20:27:44.857754 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:27:44.857479 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs podName:86b6d922-1a55-4844-b308-127fc062e056 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:46.857461762 +0000 UTC m=+252.212208079 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs") pod "network-metrics-daemon-wqxvp" (UID: "86b6d922-1a55-4844-b308-127fc062e056") : secret "metrics-daemon-secret" not found Jun 11 20:28:15.071756 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:28:15.071713 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" podUID="a491ede1-e21e-4861-8023-2a5d7d331351" Jun 11 20:28:15.091886 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:28:15.091853 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wqxvp" podUID="86b6d922-1a55-4844-b308-127fc062e056" Jun 11 20:28:15.108052 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:28:15.108018 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qpm6t" podUID="e98f77d6-3aa7-4ace-8987-93b443679091" Jun 11 20:28:15.573601 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:28:15.573576 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpm6t" Jun 11 20:28:15.573759 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:28:15.573577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:28:20.001654 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:28:20.001613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:28:20.001654 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:28:20.001661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:28:20.002233 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:28:20.001773 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 11 20:28:20.002233 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:28:20.001794 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh: secret "image-registry-tls" not found Jun 11 20:28:20.002233 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:28:20.001798 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 11 20:28:20.002233 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:28:20.001849 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls podName:a491ede1-e21e-4861-8023-2a5d7d331351 nodeName:}" failed. No retries permitted until 2026-06-11 20:30:22.001832643 +0000 UTC m=+287.356578955 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls") pod "image-registry-6bbdc7f8b7-9qvxh" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351") : secret "image-registry-tls" not found Jun 11 20:28:20.002233 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:28:20.001863 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls podName:e98f77d6-3aa7-4ace-8987-93b443679091 nodeName:}" failed. No retries permitted until 2026-06-11 20:30:22.001857088 +0000 UTC m=+287.356603382 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls") pod "dns-default-qpm6t" (UID: "e98f77d6-3aa7-4ace-8987-93b443679091") : secret "dns-default-metrics-tls" not found Jun 11 20:28:30.070766 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:28:30.070657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:29:02.403926 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.403892 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj"] Jun 11 20:29:02.406676 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.406661 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj" Jun 11 20:29:02.408858 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.408836 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jun 11 20:29:02.410094 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.410065 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jun 11 20:29:02.410276 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.410251 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wj6td\"" Jun 11 20:29:02.411252 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.411237 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24"] Jun 11 20:29:02.413892 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.413877 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5dbbb5d744-kf9lp"] Jun 11 20:29:02.414060 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.414045 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.416300 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.416264 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jun 11 20:29:02.416397 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.416302 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jun 11 20:29:02.416397 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.416267 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jun 11 20:29:02.416510 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.416495 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76bc48fcc4-kpmwf"] Jun 11 20:29:02.416609 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.416594 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jun 11 20:29:02.416668 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.416652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.417015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.416986 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tv2sn\"" Jun 11 20:29:02.419480 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.419453 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jun 11 20:29:02.419480 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.419470 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj"] Jun 11 20:29:02.419692 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.419542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.420046 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.420024 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jun 11 20:29:02.420204 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.420091 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fqqh2\"" Jun 11 20:29:02.420204 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.420130 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jun 11 20:29:02.420204 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.420157 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jun 11 20:29:02.422131 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.422084 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jun 11 20:29:02.422503 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.422326 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jun 11 20:29:02.422786 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.422662 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jun 11 20:29:02.422786 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.422759 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jun 11 20:29:02.422943 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.422789 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jun 11 20:29:02.424023 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.424001 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jun 11 20:29:02.424247 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.424230 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cl9v7\"" Jun 11 20:29:02.428472 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.428450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jun 11 20:29:02.430849 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.430830 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24"] Jun 11 20:29:02.436018 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.435993 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76bc48fcc4-kpmwf"] Jun 11 20:29:02.445009 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.444957 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5dbbb5d744-kf9lp"] Jun 11 20:29:02.501496 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501466 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98700412-c4d4-4989-8edf-90b8e43cf90d-config\") pod \"kube-storage-version-migrator-operator-7c578b78fc-lxq24\" (UID: \"98700412-c4d4-4989-8edf-90b8e43cf90d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.501496 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-default-certificate\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.501715 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.501715 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501547 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-stats-auth\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.501715 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501612 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-config\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.501715 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.501854 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvck6\" (UniqueName: \"kubernetes.io/projected/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-kube-api-access-rvck6\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.501854 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmkc\" (UniqueName: \"kubernetes.io/projected/d12a2422-a9e9-42f2-b5fe-008e194c6a0d-kube-api-access-dvmkc\") pod \"volume-data-source-validator-bd4ccd799-nk2zj\" (UID: \"d12a2422-a9e9-42f2-b5fe-008e194c6a0d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj" Jun 11 20:29:02.501854 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-trusted-ca\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.501854 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qc9\" (UniqueName: \"kubernetes.io/projected/98700412-c4d4-4989-8edf-90b8e43cf90d-kube-api-access-n6qc9\") pod \"kube-storage-version-migrator-operator-7c578b78fc-lxq24\" (UID: \"98700412-c4d4-4989-8edf-90b8e43cf90d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.501854 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4rg\" (UniqueName: \"kubernetes.io/projected/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-kube-api-access-fz4rg\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.501854 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98700412-c4d4-4989-8edf-90b8e43cf90d-serving-cert\") pod \"kube-storage-version-migrator-operator-7c578b78fc-lxq24\" (UID: \"98700412-c4d4-4989-8edf-90b8e43cf90d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.502207 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.501908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-serving-cert\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.504056 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.504035 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h"] Jun 11 20:29:02.507017 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.507001 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.508737 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.508716 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5bbd86d6bd-z825d"] Jun 11 20:29:02.509192 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.509174 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jun 11 20:29:02.509290 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.509174 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jun 11 20:29:02.509386 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.509319 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jun 11 20:29:02.509461 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.509444 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9hqlm\"" Jun 11 20:29:02.509514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.509462 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jun 11 20:29:02.511918 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.511899 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc"] Jun 11 20:29:02.512032 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.512018 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.514117 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.514097 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jun 11 20:29:02.514277 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.514251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jun 11 20:29:02.514502 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.514486 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jun 11 20:29:02.514602 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.514488 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pk5h5\"" Jun 11 20:29:02.514725 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.514709 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:02.514806 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.514777 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jun 11 20:29:02.516270 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.516250 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h"] Jun 11 20:29:02.516665 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.516648 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jun 11 20:29:02.517752 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.517732 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jun 11 20:29:02.518055 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.518034 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vmpr6\"" Jun 11 20:29:02.518194 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.518039 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jun 11 20:29:02.525341 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.525311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5bbd86d6bd-z825d"] Jun 11 20:29:02.526557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.526537 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jun 11 20:29:02.528998 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.528943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc"] Jun 11 20:29:02.601726 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.601691 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-867b76cb74-dt22k"] Jun 11 20:29:02.602689 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.602667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98700412-c4d4-4989-8edf-90b8e43cf90d-config\") pod \"kube-storage-version-migrator-operator-7c578b78fc-lxq24\" (UID: \"98700412-c4d4-4989-8edf-90b8e43cf90d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.602801 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.602701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-default-certificate\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.602801 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.602731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.602801 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.602759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-stats-auth\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.602801 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.602791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-trusted-ca-bundle\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.602990 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.602817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-config\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.602990 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:02.602851 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 11 20:29:02.602990 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:02.602926 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:03.102909312 +0000 UTC m=+208.457655611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : secret "router-metrics-certs-default" not found Jun 11 20:29:02.602990 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.602850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bf7b5a5-bb18-47ff-909c-6c3201f88856-serving-cert\") pod \"service-ca-operator-7ccbfd59dc-mf44h\" (UID: \"8bf7b5a5-bb18-47ff-909c-6c3201f88856\") " pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.603254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-serving-cert\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.603254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.603254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603104 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-tmp\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.603254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bf7b5a5-bb18-47ff-909c-6c3201f88856-config\") pod \"service-ca-operator-7ccbfd59dc-mf44h\" (UID: \"8bf7b5a5-bb18-47ff-909c-6c3201f88856\") " pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.603254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:02.603254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvck6\" (UniqueName: \"kubernetes.io/projected/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-kube-api-access-rvck6\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.603254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmkc\" (UniqueName: \"kubernetes.io/projected/d12a2422-a9e9-42f2-b5fe-008e194c6a0d-kube-api-access-dvmkc\") pod \"volume-data-source-validator-bd4ccd799-nk2zj\" (UID: \"d12a2422-a9e9-42f2-b5fe-008e194c6a0d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj" Jun 11 20:29:02.603581 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:02.603288 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:03.103247419 +0000 UTC m=+208.457993737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : configmap references non-existent config key: service-ca.crt Jun 11 20:29:02.603581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-snapshots\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.603581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-trusted-ca\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.603581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qc9\" (UniqueName: \"kubernetes.io/projected/98700412-c4d4-4989-8edf-90b8e43cf90d-kube-api-access-n6qc9\") pod \"kube-storage-version-migrator-operator-7c578b78fc-lxq24\" (UID: \"98700412-c4d4-4989-8edf-90b8e43cf90d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.603581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603438 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98700412-c4d4-4989-8edf-90b8e43cf90d-config\") pod \"kube-storage-version-migrator-operator-7c578b78fc-lxq24\" (UID: \"98700412-c4d4-4989-8edf-90b8e43cf90d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.603581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4rg\" (UniqueName: \"kubernetes.io/projected/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-kube-api-access-fz4rg\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.603581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwk5\" (UniqueName: \"kubernetes.io/projected/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-kube-api-access-scwk5\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.603581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-service-ca-bundle\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.603581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603570 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-config\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.603989 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98700412-c4d4-4989-8edf-90b8e43cf90d-serving-cert\") pod \"kube-storage-version-migrator-operator-7c578b78fc-lxq24\" (UID: \"98700412-c4d4-4989-8edf-90b8e43cf90d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.603989 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv7tn\" (UniqueName: \"kubernetes.io/projected/8bf7b5a5-bb18-47ff-909c-6c3201f88856-kube-api-access-qv7tn\") pod \"service-ca-operator-7ccbfd59dc-mf44h\" (UID: \"8bf7b5a5-bb18-47ff-909c-6c3201f88856\") " pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.603989 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn85q\" (UniqueName: \"kubernetes.io/projected/344687ec-6871-4883-b290-c89548faa703-kube-api-access-jn85q\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:02.603989 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.603784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-serving-cert\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.604479 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.604460 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-trusted-ca\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.605308 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.605278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:02.605530 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.605510 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-stats-auth\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.605803 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.605778 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98700412-c4d4-4989-8edf-90b8e43cf90d-serving-cert\") pod \"kube-storage-version-migrator-operator-7c578b78fc-lxq24\" (UID: \"98700412-c4d4-4989-8edf-90b8e43cf90d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.606158 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.606125 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-default-certificate\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.606303 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.606288 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-serving-cert\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.607245 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.607224 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jun 11 20:29:02.607336 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.607254 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jun 11 20:29:02.607336 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.607317 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mjvs6\"" Jun 11 20:29:02.614095 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.614068 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvck6\" (UniqueName: \"kubernetes.io/projected/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-kube-api-access-rvck6\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:02.614688 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.614662 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-867b76cb74-dt22k"] Jun 11 20:29:02.617267 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.617238 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qc9\" (UniqueName: \"kubernetes.io/projected/98700412-c4d4-4989-8edf-90b8e43cf90d-kube-api-access-n6qc9\") pod \"kube-storage-version-migrator-operator-7c578b78fc-lxq24\" (UID: \"98700412-c4d4-4989-8edf-90b8e43cf90d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.617637 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.617617 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4rg\" (UniqueName: \"kubernetes.io/projected/8aaeb72a-cce3-4f1c-a111-69d9c132d38e-kube-api-access-fz4rg\") pod \"console-operator-5dbbb5d744-kf9lp\" (UID: \"8aaeb72a-cce3-4f1c-a111-69d9c132d38e\") " pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.617783 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.617762 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmkc\" (UniqueName: \"kubernetes.io/projected/d12a2422-a9e9-42f2-b5fe-008e194c6a0d-kube-api-access-dvmkc\") pod \"volume-data-source-validator-bd4ccd799-nk2zj\" (UID: \"d12a2422-a9e9-42f2-b5fe-008e194c6a0d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj" Jun 11 20:29:02.704582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-tmp\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.704582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bf7b5a5-bb18-47ff-909c-6c3201f88856-config\") pod \"service-ca-operator-7ccbfd59dc-mf44h\" (UID: \"8bf7b5a5-bb18-47ff-909c-6c3201f88856\") " pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.704582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8a2c57c0-8141-4960-8b86-d8783a208d7e-nginx-conf\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:02.704582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:02.705025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-snapshots\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.705025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwk5\" (UniqueName: \"kubernetes.io/projected/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-kube-api-access-scwk5\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.705025 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:02.704698 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 11 20:29:02.705025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:02.705025 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:02.704782 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls podName:344687ec-6871-4883-b290-c89548faa703 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:03.204761051 +0000 UTC m=+208.559507350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls") pod "cluster-samples-operator-9547488fd-j8xrc" (UID: "344687ec-6871-4883-b290-c89548faa703") : secret "samples-operator-tls" not found Jun 11 20:29:02.705025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-service-ca-bundle\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.705025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv7tn\" (UniqueName: \"kubernetes.io/projected/8bf7b5a5-bb18-47ff-909c-6c3201f88856-kube-api-access-qv7tn\") pod \"service-ca-operator-7ccbfd59dc-mf44h\" (UID: \"8bf7b5a5-bb18-47ff-909c-6c3201f88856\") " pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.705025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn85q\" (UniqueName: \"kubernetes.io/projected/344687ec-6871-4883-b290-c89548faa703-kube-api-access-jn85q\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:02.705025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.704937 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-tmp\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.705491 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.705032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-trusted-ca-bundle\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.705491 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.705062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bf7b5a5-bb18-47ff-909c-6c3201f88856-serving-cert\") pod \"service-ca-operator-7ccbfd59dc-mf44h\" (UID: \"8bf7b5a5-bb18-47ff-909c-6c3201f88856\") " pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.705491 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.705084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-serving-cert\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.705491 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.705184 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bf7b5a5-bb18-47ff-909c-6c3201f88856-config\") pod \"service-ca-operator-7ccbfd59dc-mf44h\" (UID: \"8bf7b5a5-bb18-47ff-909c-6c3201f88856\") " pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.705491 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.705323 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-snapshots\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.705491 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.705371 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-service-ca-bundle\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.705889 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.705832 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-trusted-ca-bundle\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.707347 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.707321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bf7b5a5-bb18-47ff-909c-6c3201f88856-serving-cert\") pod \"service-ca-operator-7ccbfd59dc-mf44h\" (UID: \"8bf7b5a5-bb18-47ff-909c-6c3201f88856\") " pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.707467 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.707447 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-serving-cert\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.714571 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.714541 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv7tn\" (UniqueName: \"kubernetes.io/projected/8bf7b5a5-bb18-47ff-909c-6c3201f88856-kube-api-access-qv7tn\") pod \"service-ca-operator-7ccbfd59dc-mf44h\" (UID: \"8bf7b5a5-bb18-47ff-909c-6c3201f88856\") " pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.714571 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.714566 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn85q\" (UniqueName: \"kubernetes.io/projected/344687ec-6871-4883-b290-c89548faa703-kube-api-access-jn85q\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:02.714707 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.714615 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwk5\" (UniqueName: \"kubernetes.io/projected/c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f-kube-api-access-scwk5\") pod \"insights-operator-5bbd86d6bd-z825d\" (UID: \"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f\") " pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.716425 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.716410 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj" Jun 11 20:29:02.726331 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.726307 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" Jun 11 20:29:02.732332 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.732304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:02.806801 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.806424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8a2c57c0-8141-4960-8b86-d8783a208d7e-nginx-conf\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:02.806801 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.806503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:02.806801 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:02.806640 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 11 20:29:02.806801 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:02.806706 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert podName:8a2c57c0-8141-4960-8b86-d8783a208d7e nodeName:}" failed. No retries permitted until 2026-06-11 20:29:03.306687617 +0000 UTC m=+208.661433914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-dt22k" (UID: "8a2c57c0-8141-4960-8b86-d8783a208d7e") : secret "networking-console-plugin-cert" not found Jun 11 20:29:02.807510 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.807469 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8a2c57c0-8141-4960-8b86-d8783a208d7e-nginx-conf\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:02.818470 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.818442 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" Jun 11 20:29:02.828134 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.828104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" Jun 11 20:29:02.870363 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.870316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj"] Jun 11 20:29:02.874865 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:02.874805 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12a2422_a9e9_42f2_b5fe_008e194c6a0d.slice/crio-d0871b66bcf01745b301a61ef120d4d3c748ac23558994da3c735e1d400da9db WatchSource:0}: Error finding container d0871b66bcf01745b301a61ef120d4d3c748ac23558994da3c735e1d400da9db: Status 404 returned error can't find the container with id d0871b66bcf01745b301a61ef120d4d3c748ac23558994da3c735e1d400da9db Jun 11 20:29:02.961715 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.961637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h"] Jun 11 20:29:02.964653 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:02.964625 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf7b5a5_bb18_47ff_909c_6c3201f88856.slice/crio-0b8314efb7f63c68afd86726d55643a4b1d378d96c851205bcf9c68f1dee32c0 WatchSource:0}: Error finding container 0b8314efb7f63c68afd86726d55643a4b1d378d96c851205bcf9c68f1dee32c0: Status 404 returned error can't find the container with id 0b8314efb7f63c68afd86726d55643a4b1d378d96c851205bcf9c68f1dee32c0 Jun 11 20:29:02.980559 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:02.980532 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5bbd86d6bd-z825d"] Jun 11 20:29:02.983086 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:02.983064 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fa4480_6eee_4529_9ffc_c7b7b4b17c8f.slice/crio-12fb5e6911761f141843b2c282331839df391ab460fc52b914a023c12cfae62a WatchSource:0}: Error finding container 12fb5e6911761f141843b2c282331839df391ab460fc52b914a023c12cfae62a: Status 404 returned error can't find the container with id 12fb5e6911761f141843b2c282331839df391ab460fc52b914a023c12cfae62a Jun 11 20:29:03.086735 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.086698 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24"] Jun 11 20:29:03.089333 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.089308 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5dbbb5d744-kf9lp"] Jun 11 20:29:03.090404 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:03.090383 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98700412_c4d4_4989_8edf_90b8e43cf90d.slice/crio-06ad4e0b9f75f6b525e7fef6a664a4745092e7c025ce07b8a2c5ccb84bb91541 WatchSource:0}: Error finding container 06ad4e0b9f75f6b525e7fef6a664a4745092e7c025ce07b8a2c5ccb84bb91541: Status 404 returned error can't find the container with id 06ad4e0b9f75f6b525e7fef6a664a4745092e7c025ce07b8a2c5ccb84bb91541 Jun 11 20:29:03.092927 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:03.092905 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aaeb72a_cce3_4f1c_a111_69d9c132d38e.slice/crio-1eeeb031c2c671f1319e14b57e9c3a4022368f2c2e1d4502aeacd113d57ae819 WatchSource:0}: Error finding container 1eeeb031c2c671f1319e14b57e9c3a4022368f2c2e1d4502aeacd113d57ae819: Status 404 returned error can't find the container with id 1eeeb031c2c671f1319e14b57e9c3a4022368f2c2e1d4502aeacd113d57ae819 Jun 11 20:29:03.109185 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.109161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:03.109334 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.109219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:03.109334 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:03.109318 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 11 20:29:03.109441 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:03.109381 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:04.109364246 +0000 UTC m=+209.464110541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : configmap references non-existent config key: service-ca.crt Jun 11 20:29:03.109441 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:03.109404 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:04.109397358 +0000 UTC m=+209.464143652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : secret "router-metrics-certs-default" not found Jun 11 20:29:03.209848 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.209815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:03.210073 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:03.209974 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 11 20:29:03.210073 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:03.210037 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls podName:344687ec-6871-4883-b290-c89548faa703 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:04.210022895 +0000 UTC m=+209.564769190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls") pod "cluster-samples-operator-9547488fd-j8xrc" (UID: "344687ec-6871-4883-b290-c89548faa703") : secret "samples-operator-tls" not found Jun 11 20:29:03.311043 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.311004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:03.311242 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:03.311145 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 11 20:29:03.311242 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:03.311219 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert podName:8a2c57c0-8141-4960-8b86-d8783a208d7e nodeName:}" failed. No retries permitted until 2026-06-11 20:29:04.311196086 +0000 UTC m=+209.665942380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-dt22k" (UID: "8a2c57c0-8141-4960-8b86-d8783a208d7e") : secret "networking-console-plugin-cert" not found Jun 11 20:29:03.690950 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.690830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" event={"ID":"8aaeb72a-cce3-4f1c-a111-69d9c132d38e","Type":"ContainerStarted","Data":"1eeeb031c2c671f1319e14b57e9c3a4022368f2c2e1d4502aeacd113d57ae819"} Jun 11 20:29:03.692556 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.692513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj" event={"ID":"d12a2422-a9e9-42f2-b5fe-008e194c6a0d","Type":"ContainerStarted","Data":"d0871b66bcf01745b301a61ef120d4d3c748ac23558994da3c735e1d400da9db"} Jun 11 20:29:03.693905 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.693870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" event={"ID":"8bf7b5a5-bb18-47ff-909c-6c3201f88856","Type":"ContainerStarted","Data":"0b8314efb7f63c68afd86726d55643a4b1d378d96c851205bcf9c68f1dee32c0"} Jun 11 20:29:03.695443 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.695390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" event={"ID":"98700412-c4d4-4989-8edf-90b8e43cf90d","Type":"ContainerStarted","Data":"06ad4e0b9f75f6b525e7fef6a664a4745092e7c025ce07b8a2c5ccb84bb91541"} Jun 11 20:29:03.696659 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:03.696637 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" event={"ID":"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f","Type":"ContainerStarted","Data":"12fb5e6911761f141843b2c282331839df391ab460fc52b914a023c12cfae62a"} Jun 11 20:29:04.119756 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:04.119721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:04.119943 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:04.119828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:04.119943 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:04.119872 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 11 20:29:04.119943 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:04.119940 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:06.119921195 +0000 UTC m=+211.474667489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : secret "router-metrics-certs-default" not found Jun 11 20:29:04.120147 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:04.119989 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:06.11995388 +0000 UTC m=+211.474700174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : configmap references non-existent config key: service-ca.crt Jun 11 20:29:04.220685 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:04.220653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:04.220843 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:04.220815 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 11 20:29:04.220908 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:04.220883 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls podName:344687ec-6871-4883-b290-c89548faa703 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:06.220862916 +0000 UTC m=+211.575609213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls") pod "cluster-samples-operator-9547488fd-j8xrc" (UID: "344687ec-6871-4883-b290-c89548faa703") : secret "samples-operator-tls" not found Jun 11 20:29:04.322319 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:04.321742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:04.322319 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:04.321900 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 11 20:29:04.322319 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:04.321986 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert podName:8a2c57c0-8141-4960-8b86-d8783a208d7e nodeName:}" failed. No retries permitted until 2026-06-11 20:29:06.321946382 +0000 UTC m=+211.676692689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-dt22k" (UID: "8a2c57c0-8141-4960-8b86-d8783a208d7e") : secret "networking-console-plugin-cert" not found Jun 11 20:29:04.700586 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:04.700545 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj" event={"ID":"d12a2422-a9e9-42f2-b5fe-008e194c6a0d","Type":"ContainerStarted","Data":"e749da1809b49c0e13239c4a3a74abc78a4cd6ade0fd2a2118a8b5402d876fc3"} Jun 11 20:29:04.714693 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:04.714636 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nk2zj" podStartSLOduration=1.4352435350000001 podStartE2EDuration="2.71458942s" totalImagesPullingTime="1.279345885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:04.714083792 +0000 UTC m=+210.068830109" watchObservedRunningTime="2026-06-11 20:29:04.71458942 +0000 UTC m=+210.069335737" Jun 11 20:29:06.138911 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:06.138861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:06.139406 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:06.138973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:06.139406 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:06.139044 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 11 20:29:06.139406 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:06.139130 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:10.1391078 +0000 UTC m=+215.493854111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : secret "router-metrics-certs-default" not found Jun 11 20:29:06.139406 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:06.139150 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:10.139140393 +0000 UTC m=+215.493886687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : configmap references non-existent config key: service-ca.crt Jun 11 20:29:06.240126 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:06.240105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:06.240264 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:06.240246 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 11 20:29:06.240318 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:06.240309 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls podName:344687ec-6871-4883-b290-c89548faa703 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:10.240293992 +0000 UTC m=+215.595040289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls") pod "cluster-samples-operator-9547488fd-j8xrc" (UID: "344687ec-6871-4883-b290-c89548faa703") : secret "samples-operator-tls" not found Jun 11 20:29:06.341093 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:06.341047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:06.341262 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:06.341215 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 11 20:29:06.341320 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:06.341292 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert podName:8a2c57c0-8141-4960-8b86-d8783a208d7e nodeName:}" failed. No retries permitted until 2026-06-11 20:29:10.341271601 +0000 UTC m=+215.696017912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-dt22k" (UID: "8a2c57c0-8141-4960-8b86-d8783a208d7e") : secret "networking-console-plugin-cert" not found Jun 11 20:29:07.710638 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.710602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" event={"ID":"8bf7b5a5-bb18-47ff-909c-6c3201f88856","Type":"ContainerStarted","Data":"2bef83aeea795b18d723ed6f02c788623140e5d1934dea58fcc2e3f587c82672"} Jun 11 20:29:07.712271 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.712241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" event={"ID":"98700412-c4d4-4989-8edf-90b8e43cf90d","Type":"ContainerStarted","Data":"79e561f8ff4bb78a9a240da18fac2ba1526bfdf5dccad9ee5bffbfdf09b20d2a"} Jun 11 20:29:07.713715 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.713686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" event={"ID":"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f","Type":"ContainerStarted","Data":"bac2e02f0e223b8ee61caa4d396ccb4e7b5831786a774541c5f620e502d5b872"} Jun 11 20:29:07.715370 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.715344 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/0.log" Jun 11 20:29:07.715479 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.715386 2564 generic.go:356] "Generic (PLEG): container finished" podID="8aaeb72a-cce3-4f1c-a111-69d9c132d38e" containerID="9378da3240a7078d89b98a84cc9811ed54c3fd220597dd653dacdea4ccbaa1b9" exitCode=255 Jun 11 20:29:07.715479 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.715418 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" event={"ID":"8aaeb72a-cce3-4f1c-a111-69d9c132d38e","Type":"ContainerDied","Data":"9378da3240a7078d89b98a84cc9811ed54c3fd220597dd653dacdea4ccbaa1b9"} Jun 11 20:29:07.715670 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.715653 2564 scope.go:117] "RemoveContainer" containerID="9378da3240a7078d89b98a84cc9811ed54c3fd220597dd653dacdea4ccbaa1b9" Jun 11 20:29:07.725774 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.725728 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" podStartSLOduration=1.947805929 podStartE2EDuration="5.725714001s" totalImagesPullingTime="3.777908072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:07.724780541 +0000 UTC m=+213.079526856" watchObservedRunningTime="2026-06-11 20:29:07.725714001 +0000 UTC m=+213.080460316" Jun 11 20:29:07.758367 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.758304 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" podStartSLOduration=2.10446827 podStartE2EDuration="5.758279684s" totalImagesPullingTime="3.653811414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:07.757794188 +0000 UTC m=+213.112540504" watchObservedRunningTime="2026-06-11 20:29:07.758279684 +0000 UTC m=+213.113026000" Jun 11 20:29:07.771738 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:07.771523 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" podStartSLOduration=1.9680892719999998 podStartE2EDuration="5.771502404s" totalImagesPullingTime="3.803413132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:07.771465124 +0000 UTC m=+213.126211460" watchObservedRunningTime="2026-06-11 20:29:07.771502404 +0000 UTC m=+213.126248722" Jun 11 20:29:08.248730 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.248690 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cq7zt"] Jun 11 20:29:08.252344 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.252314 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.254418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.254394 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jun 11 20:29:08.254557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.254399 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jun 11 20:29:08.254557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.254485 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4p9k\"" Jun 11 20:29:08.261606 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.261579 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cq7zt"] Jun 11 20:29:08.359838 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.359807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abbfb6e9-d45b-418d-8691-9b5cbb123d13-data-volume\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.360040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.359898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.360040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.359918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.360040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.359952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abbfb6e9-d45b-418d-8691-9b5cbb123d13-crio-socket\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.360040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.360008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbx9r\" (UniqueName: \"kubernetes.io/projected/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-api-access-xbx9r\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.460482 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.460446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.460482 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.460484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.460717 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.460538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abbfb6e9-d45b-418d-8691-9b5cbb123d13-crio-socket\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.460717 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:08.460586 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jun 11 20:29:08.460717 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.460617 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abbfb6e9-d45b-418d-8691-9b5cbb123d13-crio-socket\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.460717 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:08.460630 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 11 20:29:08.460717 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:08.460652 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm podName:abbfb6e9-d45b-418d-8691-9b5cbb123d13 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:08.960632779 +0000 UTC m=+214.315379099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cq7zt" (UID: "abbfb6e9-d45b-418d-8691-9b5cbb123d13") : configmap "kube-rbac-proxy" not found Jun 11 20:29:08.460717 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:08.460682 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls podName:abbfb6e9-d45b-418d-8691-9b5cbb123d13 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:08.960667772 +0000 UTC m=+214.315414066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7zt" (UID: "abbfb6e9-d45b-418d-8691-9b5cbb123d13") : secret "insights-runtime-extractor-tls" not found Jun 11 20:29:08.460717 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.460714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbx9r\" (UniqueName: \"kubernetes.io/projected/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-api-access-xbx9r\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.461021 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.460744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abbfb6e9-d45b-418d-8691-9b5cbb123d13-data-volume\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.461061 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.461046 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abbfb6e9-d45b-418d-8691-9b5cbb123d13-data-volume\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.468141 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.468113 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbx9r\" (UniqueName: \"kubernetes.io/projected/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-api-access-xbx9r\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.721187 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.721114 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:29:08.721600 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.721581 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/0.log" Jun 11 20:29:08.721672 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.721624 2564 generic.go:356] "Generic (PLEG): container finished" podID="8aaeb72a-cce3-4f1c-a111-69d9c132d38e" containerID="db700ccc99334dc40820d15c2b25ae0cc1a478d6564112d859654d2aa28c697c" exitCode=255 Jun 11 20:29:08.721738 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.721713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" event={"ID":"8aaeb72a-cce3-4f1c-a111-69d9c132d38e","Type":"ContainerDied","Data":"db700ccc99334dc40820d15c2b25ae0cc1a478d6564112d859654d2aa28c697c"} Jun 11 20:29:08.721801 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.721787 2564 scope.go:117] "RemoveContainer" containerID="9378da3240a7078d89b98a84cc9811ed54c3fd220597dd653dacdea4ccbaa1b9" Jun 11 20:29:08.722044 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.722020 2564 scope.go:117] "RemoveContainer" containerID="db700ccc99334dc40820d15c2b25ae0cc1a478d6564112d859654d2aa28c697c" Jun 11 20:29:08.722245 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:08.722225 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5dbbb5d744-kf9lp_openshift-console-operator(8aaeb72a-cce3-4f1c-a111-69d9c132d38e)\"" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" podUID="8aaeb72a-cce3-4f1c-a111-69d9c132d38e" Jun 11 20:29:08.966313 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.966277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.966313 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:08.966312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:08.966520 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:08.966396 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jun 11 20:29:08.966520 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:08.966407 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 11 20:29:08.966520 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:08.966458 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls podName:abbfb6e9-d45b-418d-8691-9b5cbb123d13 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:09.966445298 +0000 UTC m=+215.321191592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7zt" (UID: "abbfb6e9-d45b-418d-8691-9b5cbb123d13") : secret "insights-runtime-extractor-tls" not found Jun 11 20:29:08.966520 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:08.966472 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm podName:abbfb6e9-d45b-418d-8691-9b5cbb123d13 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:09.966465878 +0000 UTC m=+215.321212171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cq7zt" (UID: "abbfb6e9-d45b-418d-8691-9b5cbb123d13") : configmap "kube-rbac-proxy" not found Jun 11 20:29:09.726149 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:09.726121 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:29:09.726656 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:09.726544 2564 scope.go:117] "RemoveContainer" containerID="db700ccc99334dc40820d15c2b25ae0cc1a478d6564112d859654d2aa28c697c" Jun 11 20:29:09.726769 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:09.726746 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5dbbb5d744-kf9lp_openshift-console-operator(8aaeb72a-cce3-4f1c-a111-69d9c132d38e)\"" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" podUID="8aaeb72a-cce3-4f1c-a111-69d9c132d38e" Jun 11 20:29:09.975084 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:09.975043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:09.975084 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:09.975083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:09.975291 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:09.975172 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jun 11 20:29:09.975291 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:09.975206 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 11 20:29:09.975291 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:09.975255 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm podName:abbfb6e9-d45b-418d-8691-9b5cbb123d13 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:11.975222241 +0000 UTC m=+217.329968535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cq7zt" (UID: "abbfb6e9-d45b-418d-8691-9b5cbb123d13") : configmap "kube-rbac-proxy" not found Jun 11 20:29:09.975291 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:09.975273 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls podName:abbfb6e9-d45b-418d-8691-9b5cbb123d13 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:11.975262143 +0000 UTC m=+217.330008449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7zt" (UID: "abbfb6e9-d45b-418d-8691-9b5cbb123d13") : secret "insights-runtime-extractor-tls" not found Jun 11 20:29:10.144943 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.144914 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5fnqb_f748b10a-2034-4330-9292-f4781f1b4359/dns-node-resolver/0.log" Jun 11 20:29:10.177087 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.177056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:10.177308 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.177132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:10.177308 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:10.177212 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 11 20:29:10.177308 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:10.177271 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:18.177256497 +0000 UTC m=+223.532002792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : secret "router-metrics-certs-default" not found Jun 11 20:29:10.177460 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:10.177322 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:18.177304783 +0000 UTC m=+223.532051079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : configmap references non-existent config key: service-ca.crt Jun 11 20:29:10.278375 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.278335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:10.278546 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:10.278458 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 11 20:29:10.278546 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:10.278508 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls podName:344687ec-6871-4883-b290-c89548faa703 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:18.278495711 +0000 UTC m=+223.633242005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls") pod "cluster-samples-operator-9547488fd-j8xrc" (UID: "344687ec-6871-4883-b290-c89548faa703") : secret "samples-operator-tls" not found Jun 11 20:29:10.379792 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.379744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:10.379996 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:10.379901 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 11 20:29:10.379996 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:10.379958 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert podName:8a2c57c0-8141-4960-8b86-d8783a208d7e nodeName:}" failed. No retries permitted until 2026-06-11 20:29:18.379944908 +0000 UTC m=+223.734691205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-dt22k" (UID: "8a2c57c0-8141-4960-8b86-d8783a208d7e") : secret "networking-console-plugin-cert" not found Jun 11 20:29:10.568206 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.568174 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7b994bc59-8ct6j"] Jun 11 20:29:10.572554 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.572537 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.574987 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.574955 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jun 11 20:29:10.575100 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.574997 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fk8n7\"" Jun 11 20:29:10.575100 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.575076 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jun 11 20:29:10.575623 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.575602 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jun 11 20:29:10.575715 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.575636 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jun 11 20:29:10.579864 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.579841 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7b994bc59-8ct6j"] Jun 11 20:29:10.682550 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.682507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8nw\" (UniqueName: \"kubernetes.io/projected/ec9dfe77-2d1f-4b8b-9e28-5c145c6da331-kube-api-access-jj8nw\") pod \"service-ca-7b994bc59-8ct6j\" (UID: \"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331\") " pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.682733 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.682583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ec9dfe77-2d1f-4b8b-9e28-5c145c6da331-signing-key\") pod \"service-ca-7b994bc59-8ct6j\" (UID: \"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331\") " pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.682733 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.682705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ec9dfe77-2d1f-4b8b-9e28-5c145c6da331-signing-cabundle\") pod \"service-ca-7b994bc59-8ct6j\" (UID: \"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331\") " pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.783644 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.783602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8nw\" (UniqueName: \"kubernetes.io/projected/ec9dfe77-2d1f-4b8b-9e28-5c145c6da331-kube-api-access-jj8nw\") pod \"service-ca-7b994bc59-8ct6j\" (UID: \"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331\") " pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.784117 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.783676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ec9dfe77-2d1f-4b8b-9e28-5c145c6da331-signing-key\") pod \"service-ca-7b994bc59-8ct6j\" (UID: \"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331\") " pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.784117 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.783739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ec9dfe77-2d1f-4b8b-9e28-5c145c6da331-signing-cabundle\") pod \"service-ca-7b994bc59-8ct6j\" (UID: \"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331\") " pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.784344 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.784325 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ec9dfe77-2d1f-4b8b-9e28-5c145c6da331-signing-cabundle\") pod \"service-ca-7b994bc59-8ct6j\" (UID: \"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331\") " pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.786273 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.786242 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ec9dfe77-2d1f-4b8b-9e28-5c145c6da331-signing-key\") pod \"service-ca-7b994bc59-8ct6j\" (UID: \"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331\") " pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.790837 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.790814 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8nw\" (UniqueName: \"kubernetes.io/projected/ec9dfe77-2d1f-4b8b-9e28-5c145c6da331-kube-api-access-jj8nw\") pod \"service-ca-7b994bc59-8ct6j\" (UID: \"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331\") " pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:10.882615 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:10.882517 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" Jun 11 20:29:11.007480 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:11.007450 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7b994bc59-8ct6j"] Jun 11 20:29:11.010280 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:11.010255 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9dfe77_2d1f_4b8b_9e28_5c145c6da331.slice/crio-b6503be7d912ccd98348c5889e71c683d801da6d2e5d41bec77bee1e5a965c3a WatchSource:0}: Error finding container b6503be7d912ccd98348c5889e71c683d801da6d2e5d41bec77bee1e5a965c3a: Status 404 returned error can't find the container with id b6503be7d912ccd98348c5889e71c683d801da6d2e5d41bec77bee1e5a965c3a Jun 11 20:29:11.346283 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:11.346256 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k58bv_c0e098fe-1a43-4895-a7e0-ab36688d5b77/node-ca/0.log" Jun 11 20:29:11.735287 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:11.735199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" event={"ID":"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331","Type":"ContainerStarted","Data":"0dd0708dc9c9d67674f5e0816ad43868d4960690ab98ce2db17d8ecb27f07a6b"} Jun 11 20:29:11.735287 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:11.735258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" event={"ID":"ec9dfe77-2d1f-4b8b-9e28-5c145c6da331","Type":"ContainerStarted","Data":"b6503be7d912ccd98348c5889e71c683d801da6d2e5d41bec77bee1e5a965c3a"} Jun 11 20:29:11.750243 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:11.750186 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7b994bc59-8ct6j" podStartSLOduration=1.750171967 podStartE2EDuration="1.750171967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:11.748430002 +0000 UTC m=+217.103176319" watchObservedRunningTime="2026-06-11 20:29:11.750171967 +0000 UTC m=+217.104918283" Jun 11 20:29:11.995425 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:11.995230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:11.995425 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:11.995277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:11.995425 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:11.995378 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jun 11 20:29:11.996091 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:11.995446 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm podName:abbfb6e9-d45b-418d-8691-9b5cbb123d13 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:15.995428696 +0000 UTC m=+221.350174990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cq7zt" (UID: "abbfb6e9-d45b-418d-8691-9b5cbb123d13") : configmap "kube-rbac-proxy" not found Jun 11 20:29:11.996091 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:11.995477 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 11 20:29:11.996091 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:11.995540 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls podName:abbfb6e9-d45b-418d-8691-9b5cbb123d13 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:15.995522163 +0000 UTC m=+221.350268473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7zt" (UID: "abbfb6e9-d45b-418d-8691-9b5cbb123d13") : secret "insights-runtime-extractor-tls" not found Jun 11 20:29:12.346078 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:12.346015 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fgbx2_01568624-0240-48ae-9f43-2bc8eb597dcd/serve-healthcheck-canary/0.log" Jun 11 20:29:12.732941 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:12.732845 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:12.732941 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:12.732913 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:12.733288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:12.733275 2564 scope.go:117] "RemoveContainer" containerID="db700ccc99334dc40820d15c2b25ae0cc1a478d6564112d859654d2aa28c697c" Jun 11 20:29:12.733450 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:12.733431 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5dbbb5d744-kf9lp_openshift-console-operator(8aaeb72a-cce3-4f1c-a111-69d9c132d38e)\"" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" podUID="8aaeb72a-cce3-4f1c-a111-69d9c132d38e" Jun 11 20:29:16.033127 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:16.033084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:16.033127 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:16.033131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:16.033538 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:16.033286 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 11 20:29:16.033538 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:16.033356 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls podName:abbfb6e9-d45b-418d-8691-9b5cbb123d13 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:24.033337116 +0000 UTC m=+229.388083415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7zt" (UID: "abbfb6e9-d45b-418d-8691-9b5cbb123d13") : secret "insights-runtime-extractor-tls" not found Jun 11 20:29:16.033735 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:16.033709 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abbfb6e9-d45b-418d-8691-9b5cbb123d13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:18.253987 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:18.253928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:18.254458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:18.254066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:18.254458 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:18.254112 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle podName:fb12d4fb-c613-4500-9bf5-f09a5f001dc4 nodeName:}" failed. No retries permitted until 2026-06-11 20:29:34.254094935 +0000 UTC m=+239.608841230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle") pod "router-default-76bc48fcc4-kpmwf" (UID: "fb12d4fb-c613-4500-9bf5-f09a5f001dc4") : configmap references non-existent config key: service-ca.crt Jun 11 20:29:18.256446 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:18.256426 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-metrics-certs\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:18.355121 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:18.355077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:18.357434 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:18.357413 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/344687ec-6871-4883-b290-c89548faa703-samples-operator-tls\") pod \"cluster-samples-operator-9547488fd-j8xrc\" (UID: \"344687ec-6871-4883-b290-c89548faa703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:18.433646 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:18.433610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" Jun 11 20:29:18.456381 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:18.456336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:18.456804 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:18.456663 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 11 20:29:18.457025 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:18.457012 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert podName:8a2c57c0-8141-4960-8b86-d8783a208d7e nodeName:}" failed. No retries permitted until 2026-06-11 20:29:34.456983587 +0000 UTC m=+239.811729895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-dt22k" (UID: "8a2c57c0-8141-4960-8b86-d8783a208d7e") : secret "networking-console-plugin-cert" not found Jun 11 20:29:18.574872 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:18.574840 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc"] Jun 11 20:29:18.759414 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:18.759378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" event={"ID":"344687ec-6871-4883-b290-c89548faa703","Type":"ContainerStarted","Data":"d029b9faece8c5826ec85fe1cec26a47ea38478841c060d721d200d41f302692"} Jun 11 20:29:20.767465 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:20.767428 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" event={"ID":"344687ec-6871-4883-b290-c89548faa703","Type":"ContainerStarted","Data":"546d05638afb5afdd057e030ba2b7913ede33e3db3db939bbddf89fc53c7014f"} Jun 11 20:29:20.767937 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:20.767495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" event={"ID":"344687ec-6871-4883-b290-c89548faa703","Type":"ContainerStarted","Data":"f83adcb9dce5fb89e954e90baa8dcd8ca4fd8e9635840013083dc5927aac62d7"} Jun 11 20:29:20.782902 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:20.782853 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9547488fd-j8xrc" podStartSLOduration=17.079434825 podStartE2EDuration="18.782837194s" totalImagesPullingTime="1.703402369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:20.781944513 +0000 UTC m=+226.136690830" watchObservedRunningTime="2026-06-11 20:29:20.782837194 +0000 UTC m=+226.137583510" Jun 11 20:29:24.100902 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:24.100849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:24.103301 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:24.103276 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abbfb6e9-d45b-418d-8691-9b5cbb123d13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7zt\" (UID: \"abbfb6e9-d45b-418d-8691-9b5cbb123d13\") " pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:24.164304 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:24.164269 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cq7zt" Jun 11 20:29:24.291786 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:24.291751 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cq7zt"] Jun 11 20:29:24.294685 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:24.294661 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbfb6e9_d45b_418d_8691_9b5cbb123d13.slice/crio-2c3f2724aebc74aac4072d0683c11fc6cc305fcbf1b26ace2c863bea23ecc4d7 WatchSource:0}: Error finding container 2c3f2724aebc74aac4072d0683c11fc6cc305fcbf1b26ace2c863bea23ecc4d7: Status 404 returned error can't find the container with id 2c3f2724aebc74aac4072d0683c11fc6cc305fcbf1b26ace2c863bea23ecc4d7 Jun 11 20:29:24.781208 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:24.781169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq7zt" event={"ID":"abbfb6e9-d45b-418d-8691-9b5cbb123d13","Type":"ContainerStarted","Data":"2c3f2724aebc74aac4072d0683c11fc6cc305fcbf1b26ace2c863bea23ecc4d7"} Jun 11 20:29:27.070500 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:27.070464 2564 scope.go:117] "RemoveContainer" containerID="db700ccc99334dc40820d15c2b25ae0cc1a478d6564112d859654d2aa28c697c" Jun 11 20:29:27.793834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:27.793794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq7zt" event={"ID":"abbfb6e9-d45b-418d-8691-9b5cbb123d13","Type":"ContainerStarted","Data":"cd808b62b77e639d64aeaaf4a85983fd63607b694552b994c8ca36f0090be92f"} Jun 11 20:29:27.795636 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:27.795615 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:29:27.795760 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:27.795658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" event={"ID":"8aaeb72a-cce3-4f1c-a111-69d9c132d38e","Type":"ContainerStarted","Data":"7e482a12f6e2576e40c3ae101fe11cf58f630c4b6b029066aaca7e31a50f8b99"} Jun 11 20:29:27.795931 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:27.795913 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:27.811874 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:27.811827 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" podStartSLOduration=22.167295205 podStartE2EDuration="25.81180868s" totalImagesPullingTime="3.644513475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:27.81062458 +0000 UTC m=+233.165370896" watchObservedRunningTime="2026-06-11 20:29:27.81180868 +0000 UTC m=+233.166554996" Jun 11 20:29:28.519188 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:28.518916 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5dbbb5d744-kf9lp" Jun 11 20:29:28.800511 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:28.800476 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq7zt" event={"ID":"abbfb6e9-d45b-418d-8691-9b5cbb123d13","Type":"ContainerStarted","Data":"5e09c0c49c69975069e7cee2469e776e383d192afe6eba3ccbba898ccaaf1023"} Jun 11 20:29:30.810902 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:30.810871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq7zt" event={"ID":"abbfb6e9-d45b-418d-8691-9b5cbb123d13","Type":"ContainerStarted","Data":"818cf6bd99ca8bff9a8736e5c63725a0c9a88ec01dffb2ce845d9d277426df28"} Jun 11 20:29:30.827209 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:30.827161 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cq7zt" podStartSLOduration=17.136306913 podStartE2EDuration="22.827145651s" totalImagesPullingTime="5.690838738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:29:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:30.82664691 +0000 UTC m=+236.181393226" watchObservedRunningTime="2026-06-11 20:29:30.827145651 +0000 UTC m=+236.181891966" Jun 11 20:29:34.287078 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.287021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:34.287638 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.287616 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb12d4fb-c613-4500-9bf5-f09a5f001dc4-service-ca-bundle\") pod \"router-default-76bc48fcc4-kpmwf\" (UID: \"fb12d4fb-c613-4500-9bf5-f09a5f001dc4\") " pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:34.489114 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.489057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:34.491502 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.491479 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2c57c0-8141-4960-8b86-d8783a208d7e-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-dt22k\" (UID: \"8a2c57c0-8141-4960-8b86-d8783a208d7e\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:34.538526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.538430 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:34.690328 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.690271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76bc48fcc4-kpmwf"] Jun 11 20:29:34.692483 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:34.692457 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb12d4fb_c613_4500_9bf5_f09a5f001dc4.slice/crio-f611b16320ad3efd92b5dc7e5036c3700325fa30baf7e40b115e127916f3f7e6 WatchSource:0}: Error finding container f611b16320ad3efd92b5dc7e5036c3700325fa30baf7e40b115e127916f3f7e6: Status 404 returned error can't find the container with id f611b16320ad3efd92b5dc7e5036c3700325fa30baf7e40b115e127916f3f7e6 Jun 11 20:29:34.733660 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.733628 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" Jun 11 20:29:34.827187 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.827153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" event={"ID":"fb12d4fb-c613-4500-9bf5-f09a5f001dc4","Type":"ContainerStarted","Data":"68f1f62d06ddac00d7f8f84f8e3bc318bbb8af2902ca09e5890f24e1ec3ca44e"} Jun 11 20:29:34.827376 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.827204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" event={"ID":"fb12d4fb-c613-4500-9bf5-f09a5f001dc4","Type":"ContainerStarted","Data":"f611b16320ad3efd92b5dc7e5036c3700325fa30baf7e40b115e127916f3f7e6"} Jun 11 20:29:34.846214 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.846153 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" podStartSLOduration=32.84612603 podStartE2EDuration="32.84612603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:34.845195969 +0000 UTC m=+240.199942331" watchObservedRunningTime="2026-06-11 20:29:34.84612603 +0000 UTC m=+240.200872346" Jun 11 20:29:34.864413 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:34.864373 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-867b76cb74-dt22k"] Jun 11 20:29:34.866888 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:34.866861 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2c57c0_8141_4960_8b86_d8783a208d7e.slice/crio-3415199f18168ded22df1ad05994f2fd10293c0c131e93e90ab9ba1d3e24e8d0 WatchSource:0}: Error finding container 3415199f18168ded22df1ad05994f2fd10293c0c131e93e90ab9ba1d3e24e8d0: Status 404 returned error can't find the container with id 3415199f18168ded22df1ad05994f2fd10293c0c131e93e90ab9ba1d3e24e8d0 Jun 11 20:29:35.538859 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:35.538825 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:35.542081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:35.542057 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:35.831554 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:35.831458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" event={"ID":"8a2c57c0-8141-4960-8b86-d8783a208d7e","Type":"ContainerStarted","Data":"3415199f18168ded22df1ad05994f2fd10293c0c131e93e90ab9ba1d3e24e8d0"} Jun 11 20:29:35.831721 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:35.831648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:35.832850 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:35.832829 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76bc48fcc4-kpmwf" Jun 11 20:29:36.836475 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:36.836436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" event={"ID":"8a2c57c0-8141-4960-8b86-d8783a208d7e","Type":"ContainerStarted","Data":"4771117bf7146b3945dec03a15ce8f63383d4a087acc465bc2dd7290af007fb5"} Jun 11 20:29:36.851084 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:36.851037 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-867b76cb74-dt22k" podStartSLOduration=33.270684717 podStartE2EDuration="34.851023209s" totalImagesPullingTime="1.580338492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:36.850187474 +0000 UTC m=+242.204933790" watchObservedRunningTime="2026-06-11 20:29:36.851023209 +0000 UTC m=+242.205769525" Jun 11 20:29:37.109200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:37.109107 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h"] Jun 11 20:29:37.113066 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:37.113047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" Jun 11 20:29:37.115132 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:37.115111 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jun 11 20:29:37.115250 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:37.115144 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-f5tr6\"" Jun 11 20:29:37.118243 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:37.118221 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h"] Jun 11 20:29:37.211235 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:37.211200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f55f499-98f4-46dd-b574-9dbaf61c8c3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-krq5h\" (UID: \"5f55f499-98f4-46dd-b574-9dbaf61c8c3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" Jun 11 20:29:37.311610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:37.311566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f55f499-98f4-46dd-b574-9dbaf61c8c3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-krq5h\" (UID: \"5f55f499-98f4-46dd-b574-9dbaf61c8c3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" Jun 11 20:29:37.311800 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:37.311707 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jun 11 20:29:37.311800 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:37.311776 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f55f499-98f4-46dd-b574-9dbaf61c8c3a-tls-certificates podName:5f55f499-98f4-46dd-b574-9dbaf61c8c3a nodeName:}" failed. No retries permitted until 2026-06-11 20:29:37.811760045 +0000 UTC m=+243.166506339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/5f55f499-98f4-46dd-b574-9dbaf61c8c3a-tls-certificates") pod "prometheus-operator-admission-webhook-5cf667749c-krq5h" (UID: "5f55f499-98f4-46dd-b574-9dbaf61c8c3a") : secret "prometheus-operator-admission-webhook-tls" not found Jun 11 20:29:37.815515 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:37.815476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f55f499-98f4-46dd-b574-9dbaf61c8c3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-krq5h\" (UID: \"5f55f499-98f4-46dd-b574-9dbaf61c8c3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" Jun 11 20:29:37.817916 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:37.817896 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f55f499-98f4-46dd-b574-9dbaf61c8c3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-krq5h\" (UID: \"5f55f499-98f4-46dd-b574-9dbaf61c8c3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" Jun 11 20:29:38.023311 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:38.023263 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" Jun 11 20:29:38.151143 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:38.151116 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h"] Jun 11 20:29:38.153731 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:38.153700 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f55f499_98f4_46dd_b574_9dbaf61c8c3a.slice/crio-329b398fab305b7909166797a5ec37a31dd3355dce609d3b0610e13d67cbf23a WatchSource:0}: Error finding container 329b398fab305b7909166797a5ec37a31dd3355dce609d3b0610e13d67cbf23a: Status 404 returned error can't find the container with id 329b398fab305b7909166797a5ec37a31dd3355dce609d3b0610e13d67cbf23a Jun 11 20:29:38.845938 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:38.845897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" event={"ID":"5f55f499-98f4-46dd-b574-9dbaf61c8c3a","Type":"ContainerStarted","Data":"329b398fab305b7909166797a5ec37a31dd3355dce609d3b0610e13d67cbf23a"} Jun 11 20:29:40.855113 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:40.855080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" event={"ID":"5f55f499-98f4-46dd-b574-9dbaf61c8c3a","Type":"ContainerStarted","Data":"43dab03352b6c297e0a9b723517b04b5334a656b951ebc91d532dd27abb530d3"} Jun 11 20:29:40.855570 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:40.855311 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" Jun 11 20:29:40.859822 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:40.859802 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" Jun 11 20:29:40.868994 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:40.868924 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-krq5h" podStartSLOduration=2.277313363 podStartE2EDuration="3.868908109s" totalImagesPullingTime="1.591594746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:40.868596756 +0000 UTC m=+246.223343097" watchObservedRunningTime="2026-06-11 20:29:40.868908109 +0000 UTC m=+246.223654426" Jun 11 20:29:46.892698 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:46.892654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:29:46.895096 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:46.895072 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b6d922-1a55-4844-b308-127fc062e056-metrics-certs\") pod \"network-metrics-daemon-wqxvp\" (UID: \"86b6d922-1a55-4844-b308-127fc062e056\") " pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:29:47.173603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:47.173516 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9v6lj\"" Jun 11 20:29:47.182171 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:47.182148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqxvp" Jun 11 20:29:47.308631 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:47.308598 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wqxvp"] Jun 11 20:29:47.311681 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:47.311654 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b6d922_1a55_4844_b308_127fc062e056.slice/crio-d70d167b55f93cbb35763a05e4d904ecb4ec9f34b9fd84a5a66c68ad7f92cc8b WatchSource:0}: Error finding container d70d167b55f93cbb35763a05e4d904ecb4ec9f34b9fd84a5a66c68ad7f92cc8b: Status 404 returned error can't find the container with id d70d167b55f93cbb35763a05e4d904ecb4ec9f34b9fd84a5a66c68ad7f92cc8b Jun 11 20:29:47.878250 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:47.878205 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wqxvp" event={"ID":"86b6d922-1a55-4844-b308-127fc062e056","Type":"ContainerStarted","Data":"d70d167b55f93cbb35763a05e4d904ecb4ec9f34b9fd84a5a66c68ad7f92cc8b"} Jun 11 20:29:48.883549 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:48.883510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wqxvp" event={"ID":"86b6d922-1a55-4844-b308-127fc062e056","Type":"ContainerStarted","Data":"3da9c7f5e070927a4209d0067406219b3a4e415a678d66adeae7dad0213481d9"} Jun 11 20:29:48.883919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:48.883567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wqxvp" event={"ID":"86b6d922-1a55-4844-b308-127fc062e056","Type":"ContainerStarted","Data":"0018c33bf2b64bf935de29d36b5ecc45287c099d73f7af8df70a15dfa2a09321"} Jun 11 20:29:48.898137 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:48.898085 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wqxvp" podStartSLOduration=252.792870192 podStartE2EDuration="4m13.898068865s" totalImagesPullingTime="1.105198673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:48.897898434 +0000 UTC m=+254.252644750" watchObservedRunningTime="2026-06-11 20:29:48.898068865 +0000 UTC m=+254.252815181" Jun 11 20:29:50.487107 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.487067 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp"] Jun 11 20:29:50.489786 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.489763 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.492985 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.492942 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jun 11 20:29:50.493108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.492986 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jun 11 20:29:50.493108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.493015 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tr9vq\"" Jun 11 20:29:50.493108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.492955 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jun 11 20:29:50.493266 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.492952 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jun 11 20:29:50.493266 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.493218 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jun 11 20:29:50.501733 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.501708 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp"] Jun 11 20:29:50.514725 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.514699 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qkqtr"] Jun 11 20:29:50.517172 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.517153 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.519457 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.519302 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jun 11 20:29:50.519457 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.519331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jun 11 20:29:50.519457 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.519337 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dvlr5\"" Jun 11 20:29:50.519457 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.519357 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jun 11 20:29:50.623690 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.623654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c70e158e-548b-44a4-969d-d701a609aa7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.623690 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.623700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-tls\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.623941 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.623795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c70e158e-548b-44a4-969d-d701a609aa7f-metrics-client-ca\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.623941 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.623868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/933a7689-fe60-4573-ba8c-ec03a820295f-metrics-client-ca\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.623941 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.623894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/933a7689-fe60-4573-ba8c-ec03a820295f-sys\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.624096 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.623940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.624096 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.623991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.624096 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.624021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmt8\" (UniqueName: \"kubernetes.io/projected/c70e158e-548b-44a4-969d-d701a609aa7f-kube-api-access-7fmt8\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.624096 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.624045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqdj9\" (UniqueName: \"kubernetes.io/projected/933a7689-fe60-4573-ba8c-ec03a820295f-kube-api-access-lqdj9\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.624234 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.624104 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-textfile\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.624234 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.624145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-wtmp\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.624234 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.624174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/933a7689-fe60-4573-ba8c-ec03a820295f-root\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.624234 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.624206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70e158e-548b-44a4-969d-d701a609aa7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.725452 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.725417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70e158e-548b-44a4-969d-d701a609aa7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.725626 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.725547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c70e158e-548b-44a4-969d-d701a609aa7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.725626 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.725587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-tls\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.725730 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.725642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c70e158e-548b-44a4-969d-d701a609aa7f-metrics-client-ca\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.725782 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:50.725768 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jun 11 20:29:50.725830 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.725806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/933a7689-fe60-4573-ba8c-ec03a820295f-metrics-client-ca\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.725880 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:29:50.725845 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-tls podName:933a7689-fe60-4573-ba8c-ec03a820295f nodeName:}" failed. No retries permitted until 2026-06-11 20:29:51.225816438 +0000 UTC m=+256.580562752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-tls") pod "node-exporter-qkqtr" (UID: "933a7689-fe60-4573-ba8c-ec03a820295f") : secret "node-exporter-tls" not found Jun 11 20:29:50.726033 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/933a7689-fe60-4573-ba8c-ec03a820295f-sys\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726162 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726162 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726162 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726116 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/933a7689-fe60-4573-ba8c-ec03a820295f-sys\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726162 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmt8\" (UniqueName: \"kubernetes.io/projected/c70e158e-548b-44a4-969d-d701a609aa7f-kube-api-access-7fmt8\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.726162 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqdj9\" (UniqueName: \"kubernetes.io/projected/933a7689-fe60-4573-ba8c-ec03a820295f-kube-api-access-lqdj9\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726421 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-textfile\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726421 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-wtmp\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726421 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/933a7689-fe60-4573-ba8c-ec03a820295f-root\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726421 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726352 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/933a7689-fe60-4573-ba8c-ec03a820295f-root\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726421 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726395 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c70e158e-548b-44a4-969d-d701a609aa7f-metrics-client-ca\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.726683 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726507 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/933a7689-fe60-4573-ba8c-ec03a820295f-metrics-client-ca\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726683 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726661 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-wtmp\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726790 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726758 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-textfile\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.726842 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.726797 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.728449 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.728421 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c70e158e-548b-44a4-969d-d701a609aa7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.728449 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.728438 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c70e158e-548b-44a4-969d-d701a609aa7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.728710 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.728688 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.737272 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.737214 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmt8\" (UniqueName: \"kubernetes.io/projected/c70e158e-548b-44a4-969d-d701a609aa7f-kube-api-access-7fmt8\") pod \"openshift-state-metrics-6b8d965dd5-bjkkp\" (UID: \"c70e158e-548b-44a4-969d-d701a609aa7f\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.737560 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.737543 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqdj9\" (UniqueName: \"kubernetes.io/projected/933a7689-fe60-4573-ba8c-ec03a820295f-kube-api-access-lqdj9\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:50.802016 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.801975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" Jun 11 20:29:50.942283 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:50.942255 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp"] Jun 11 20:29:50.944146 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:50.944105 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70e158e_548b_44a4_969d_d701a609aa7f.slice/crio-383818174f4373fa4a7444c15f0a26164ad562bb92db813d2f8b57e8f568c8ea WatchSource:0}: Error finding container 383818174f4373fa4a7444c15f0a26164ad562bb92db813d2f8b57e8f568c8ea: Status 404 returned error can't find the container with id 383818174f4373fa4a7444c15f0a26164ad562bb92db813d2f8b57e8f568c8ea Jun 11 20:29:51.231469 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:51.231422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-tls\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:51.233855 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:51.233821 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/933a7689-fe60-4573-ba8c-ec03a820295f-node-exporter-tls\") pod \"node-exporter-qkqtr\" (UID: \"933a7689-fe60-4573-ba8c-ec03a820295f\") " pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:51.429135 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:51.429103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qkqtr" Jun 11 20:29:51.440911 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:51.440879 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933a7689_fe60_4573_ba8c_ec03a820295f.slice/crio-b0c1952e711720028c373f8c66364c257d6d0b6f49643a07d58f6e98875ea4ed WatchSource:0}: Error finding container b0c1952e711720028c373f8c66364c257d6d0b6f49643a07d58f6e98875ea4ed: Status 404 returned error can't find the container with id b0c1952e711720028c373f8c66364c257d6d0b6f49643a07d58f6e98875ea4ed Jun 11 20:29:51.899063 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:51.899023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" event={"ID":"c70e158e-548b-44a4-969d-d701a609aa7f","Type":"ContainerStarted","Data":"ec2dfc0a13593f4b6e4bb94506832f4badc8e657b6e682567e1da0434386c4b8"} Jun 11 20:29:51.899518 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:51.899104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" event={"ID":"c70e158e-548b-44a4-969d-d701a609aa7f","Type":"ContainerStarted","Data":"f816a01635c58da16b37c08622e847a42fe034002ce89eff1b424f6d12ca273b"} Jun 11 20:29:51.899518 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:51.899135 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" event={"ID":"c70e158e-548b-44a4-969d-d701a609aa7f","Type":"ContainerStarted","Data":"383818174f4373fa4a7444c15f0a26164ad562bb92db813d2f8b57e8f568c8ea"} Jun 11 20:29:51.900625 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:51.900591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkqtr" event={"ID":"933a7689-fe60-4573-ba8c-ec03a820295f","Type":"ContainerStarted","Data":"b0c1952e711720028c373f8c66364c257d6d0b6f49643a07d58f6e98875ea4ed"} Jun 11 20:29:52.476021 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.475940 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m"] Jun 11 20:29:52.479883 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.479855 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.483025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.482791 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jun 11 20:29:52.483025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.482791 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jun 11 20:29:52.483025 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.482867 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jun 11 20:29:52.483638 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.483617 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jun 11 20:29:52.483752 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.483727 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jun 11 20:29:52.484194 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.484182 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-kfrmx\"" Jun 11 20:29:52.484253 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.484214 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9hk10eiaer3rr\"" Jun 11 20:29:52.491480 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.491442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m"] Jun 11 20:29:52.542539 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.542503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-tls\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.542539 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.542538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.542731 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.542569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.542731 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.542615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a96f5138-1a26-4752-a96c-414575e9d99f-metrics-client-ca\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.542731 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.542706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twp47\" (UniqueName: \"kubernetes.io/projected/a96f5138-1a26-4752-a96c-414575e9d99f-kube-api-access-twp47\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.542838 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.542751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.542838 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.542789 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.542937 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.542895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-grpc-tls\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.643616 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.643534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.643616 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.643586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.643797 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.643628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-grpc-tls\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.643797 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.643652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-tls\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.643797 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.643671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.643933 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.643827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.643933 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.643880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a96f5138-1a26-4752-a96c-414575e9d99f-metrics-client-ca\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.644052 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.643935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twp47\" (UniqueName: \"kubernetes.io/projected/a96f5138-1a26-4752-a96c-414575e9d99f-kube-api-access-twp47\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.644829 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.644797 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a96f5138-1a26-4752-a96c-414575e9d99f-metrics-client-ca\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.646592 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.646566 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.646708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.646604 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.646780 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.646729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.646924 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.646902 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-tls\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.646999 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.646906 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.646999 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.646985 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a96f5138-1a26-4752-a96c-414575e9d99f-secret-grpc-tls\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.651207 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.651189 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twp47\" (UniqueName: \"kubernetes.io/projected/a96f5138-1a26-4752-a96c-414575e9d99f-kube-api-access-twp47\") pod \"thanos-querier-57c8cffbc7-8jq9m\" (UID: \"a96f5138-1a26-4752-a96c-414575e9d99f\") " pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.790820 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.790779 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:52.908205 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.908127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" event={"ID":"c70e158e-548b-44a4-969d-d701a609aa7f","Type":"ContainerStarted","Data":"b133174ade6b425d3df0556b2cf9b1447ca2e27726353d6cbd5b01f7634c558d"} Jun 11 20:29:52.909553 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.909525 2564 generic.go:356] "Generic (PLEG): container finished" podID="933a7689-fe60-4573-ba8c-ec03a820295f" containerID="ac200f80a451841c5df899539740ae91677a897e38eb9a1fef741ca23a454898" exitCode=0 Jun 11 20:29:52.909657 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.909587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkqtr" event={"ID":"933a7689-fe60-4573-ba8c-ec03a820295f","Type":"ContainerDied","Data":"ac200f80a451841c5df899539740ae91677a897e38eb9a1fef741ca23a454898"} Jun 11 20:29:52.929638 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.929607 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m"] Jun 11 20:29:52.930527 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:52.930479 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-bjkkp" podStartSLOduration=1.629482887 podStartE2EDuration="2.930463527s" totalImagesPullingTime="1.30098064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:52.929056683 +0000 UTC m=+258.283803011" watchObservedRunningTime="2026-06-11 20:29:52.930463527 +0000 UTC m=+258.285209844" Jun 11 20:29:52.934000 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:52.933955 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96f5138_1a26_4752_a96c_414575e9d99f.slice/crio-dbd0a4cf1c7cc2e4bd49ed8ae14c90aff5a56675f1b9340e6f2be13c17166a49 WatchSource:0}: Error finding container dbd0a4cf1c7cc2e4bd49ed8ae14c90aff5a56675f1b9340e6f2be13c17166a49: Status 404 returned error can't find the container with id dbd0a4cf1c7cc2e4bd49ed8ae14c90aff5a56675f1b9340e6f2be13c17166a49 Jun 11 20:29:53.915150 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:53.915100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" event={"ID":"a96f5138-1a26-4752-a96c-414575e9d99f","Type":"ContainerStarted","Data":"dbd0a4cf1c7cc2e4bd49ed8ae14c90aff5a56675f1b9340e6f2be13c17166a49"} Jun 11 20:29:53.917810 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:53.917782 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkqtr" event={"ID":"933a7689-fe60-4573-ba8c-ec03a820295f","Type":"ContainerStarted","Data":"5278b1296b9e0ef19aa6e4e37c96e0f0480f526baea67e17675190377ca26ac3"} Jun 11 20:29:53.917984 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:53.917841 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkqtr" event={"ID":"933a7689-fe60-4573-ba8c-ec03a820295f","Type":"ContainerStarted","Data":"2580cd865121996831371afb0710ec5b776335569b1af0ca176ea3013cca93b5"} Jun 11 20:29:53.936783 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:53.936716 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qkqtr" podStartSLOduration=2.944717206 podStartE2EDuration="3.93670024s" totalImagesPullingTime="918.340839ms" totalInitContainerRuntime="73.642195ms" isStatefulPod=true podCreationTimestamp="2026-06-11 20:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:53.934934423 +0000 UTC m=+259.289680803" watchObservedRunningTime="2026-06-11 20:29:53.93670024 +0000 UTC m=+259.291446574" Jun 11 20:29:54.899164 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.899126 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84b49b6d67-9dvrm"] Jun 11 20:29:54.903186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.903162 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:54.905442 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.905415 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jun 11 20:29:54.906206 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.906175 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vdqkg\"" Jun 11 20:29:54.906206 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.906175 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jun 11 20:29:54.906354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.906213 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jun 11 20:29:54.906354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.906180 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-76ckkr8molm01\"" Jun 11 20:29:54.906354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.906317 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jun 11 20:29:54.912069 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.911940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84b49b6d67-9dvrm"] Jun 11 20:29:54.966163 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.966120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/24a55637-1ee5-4825-9a7b-2cae9e594ac5-secret-metrics-server-client-certs\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:54.966582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.966181 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/24a55637-1ee5-4825-9a7b-2cae9e594ac5-metrics-server-audit-profiles\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:54.966582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.966218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/24a55637-1ee5-4825-9a7b-2cae9e594ac5-secret-metrics-server-tls\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:54.966582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.966253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24a55637-1ee5-4825-9a7b-2cae9e594ac5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:54.966582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.966326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqhc\" (UniqueName: \"kubernetes.io/projected/24a55637-1ee5-4825-9a7b-2cae9e594ac5-kube-api-access-grqhc\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:54.966582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.966381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/24a55637-1ee5-4825-9a7b-2cae9e594ac5-client-ca-bundle\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:54.966582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:54.966407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/24a55637-1ee5-4825-9a7b-2cae9e594ac5-audit-log\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.067589 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.067551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/24a55637-1ee5-4825-9a7b-2cae9e594ac5-metrics-server-audit-profiles\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.067589 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.067595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/24a55637-1ee5-4825-9a7b-2cae9e594ac5-secret-metrics-server-tls\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.067872 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.067618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24a55637-1ee5-4825-9a7b-2cae9e594ac5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.067872 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.067674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqhc\" (UniqueName: \"kubernetes.io/projected/24a55637-1ee5-4825-9a7b-2cae9e594ac5-kube-api-access-grqhc\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.067872 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.067709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/24a55637-1ee5-4825-9a7b-2cae9e594ac5-client-ca-bundle\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.068546 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.068009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/24a55637-1ee5-4825-9a7b-2cae9e594ac5-audit-log\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.068546 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.068133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/24a55637-1ee5-4825-9a7b-2cae9e594ac5-secret-metrics-server-client-certs\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.068546 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.068436 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24a55637-1ee5-4825-9a7b-2cae9e594ac5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.068546 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.068439 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/24a55637-1ee5-4825-9a7b-2cae9e594ac5-audit-log\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.068809 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.068774 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/24a55637-1ee5-4825-9a7b-2cae9e594ac5-metrics-server-audit-profiles\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.070850 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.070824 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/24a55637-1ee5-4825-9a7b-2cae9e594ac5-secret-metrics-server-client-certs\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.070992 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.070882 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/24a55637-1ee5-4825-9a7b-2cae9e594ac5-client-ca-bundle\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.071100 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.071078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/24a55637-1ee5-4825-9a7b-2cae9e594ac5-secret-metrics-server-tls\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.076690 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.076666 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqhc\" (UniqueName: \"kubernetes.io/projected/24a55637-1ee5-4825-9a7b-2cae9e594ac5-kube-api-access-grqhc\") pod \"metrics-server-84b49b6d67-9dvrm\" (UID: \"24a55637-1ee5-4825-9a7b-2cae9e594ac5\") " pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.215316 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.215231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:29:55.323340 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.323300 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6954888546-vcghk"] Jun 11 20:29:55.327086 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.327046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.329923 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.329763 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jun 11 20:29:55.329923 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.329782 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8mmxj\"" Jun 11 20:29:55.329923 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.329763 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jun 11 20:29:55.329923 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.329817 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jun 11 20:29:55.330268 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.330229 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jun 11 20:29:55.330517 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.330499 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jun 11 20:29:55.330737 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.330718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jun 11 20:29:55.330977 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.330944 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jun 11 20:29:55.334617 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.334594 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jun 11 20:29:55.338578 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.338554 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6954888546-vcghk"] Jun 11 20:29:55.471327 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.471258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-oauth-serving-cert\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.471327 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.471298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-serving-cert\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.471521 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.471335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-config\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.471521 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.471436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-service-ca\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.471521 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.471478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-oauth-config\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.471662 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.471521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-trusted-ca-bundle\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.471662 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.471547 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbgvd\" (UniqueName: \"kubernetes.io/projected/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-kube-api-access-tbgvd\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.494937 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.494917 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84b49b6d67-9dvrm"] Jun 11 20:29:55.497159 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:55.497133 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a55637_1ee5_4825_9a7b_2cae9e594ac5.slice/crio-01cabb23c920981fe891dcecfe7bc1788304a439b3953948b9a14aa83d314205 WatchSource:0}: Error finding container 01cabb23c920981fe891dcecfe7bc1788304a439b3953948b9a14aa83d314205: Status 404 returned error can't find the container with id 01cabb23c920981fe891dcecfe7bc1788304a439b3953948b9a14aa83d314205 Jun 11 20:29:55.572134 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.572105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-config\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.572261 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.572159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-service-ca\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.572261 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.572198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-oauth-config\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.572261 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.572239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-trusted-ca-bundle\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.572412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.572261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbgvd\" (UniqueName: \"kubernetes.io/projected/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-kube-api-access-tbgvd\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.572412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.572340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-oauth-serving-cert\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.572412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.572369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-serving-cert\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.572986 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.572842 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-config\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.573317 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.573293 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-service-ca\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.573453 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.573433 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-oauth-serving-cert\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.573892 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.573867 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-trusted-ca-bundle\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.575871 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.575838 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-oauth-config\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.576335 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.576315 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-serving-cert\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.579847 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.579826 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbgvd\" (UniqueName: \"kubernetes.io/projected/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-kube-api-access-tbgvd\") pod \"console-6954888546-vcghk\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.640293 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.640264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6954888546-vcghk" Jun 11 20:29:55.774110 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.774070 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6954888546-vcghk"] Jun 11 20:29:55.777223 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:29:55.777194 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ad3810_fbf7_41b9_814a_7e4c1d5b812d.slice/crio-524a5fbd86aba709940c1450cf1fd334fb1e3f677a10b21a30074aa70de850ac WatchSource:0}: Error finding container 524a5fbd86aba709940c1450cf1fd334fb1e3f677a10b21a30074aa70de850ac: Status 404 returned error can't find the container with id 524a5fbd86aba709940c1450cf1fd334fb1e3f677a10b21a30074aa70de850ac Jun 11 20:29:55.927007 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.926974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6954888546-vcghk" event={"ID":"43ad3810-fbf7-41b9-814a-7e4c1d5b812d","Type":"ContainerStarted","Data":"524a5fbd86aba709940c1450cf1fd334fb1e3f677a10b21a30074aa70de850ac"} Jun 11 20:29:55.928046 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.928026 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" event={"ID":"24a55637-1ee5-4825-9a7b-2cae9e594ac5","Type":"ContainerStarted","Data":"01cabb23c920981fe891dcecfe7bc1788304a439b3953948b9a14aa83d314205"} Jun 11 20:29:55.930085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.930067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" event={"ID":"a96f5138-1a26-4752-a96c-414575e9d99f","Type":"ContainerStarted","Data":"fa8452b3635dd86968dd198ab7ec20a9176b620e799312f780e350a6febcfa7e"} Jun 11 20:29:55.930183 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.930115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" event={"ID":"a96f5138-1a26-4752-a96c-414575e9d99f","Type":"ContainerStarted","Data":"79479d7cd874e9ad7a2179141937d57d5e3121d70d0391dc28b8f024a2a9d65e"} Jun 11 20:29:55.930183 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:55.930146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" event={"ID":"a96f5138-1a26-4752-a96c-414575e9d99f","Type":"ContainerStarted","Data":"5915484bd988fce5be95914705db404a02252cbe3b1870efc7816c61c8267302"} Jun 11 20:29:56.939291 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:56.939252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" event={"ID":"a96f5138-1a26-4752-a96c-414575e9d99f","Type":"ContainerStarted","Data":"f9d435b28d40fa5dbf171ddf47ddac27ec2728b0d3b0f8a4ee72c2a3ed89f21e"} Jun 11 20:29:56.939813 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:56.939375 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" event={"ID":"a96f5138-1a26-4752-a96c-414575e9d99f","Type":"ContainerStarted","Data":"d2b2f3dcf9487ec39bfde77a347beaf46c6899c86ae827d104790b065b9c52fd"} Jun 11 20:29:56.939813 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:56.939421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" event={"ID":"a96f5138-1a26-4752-a96c-414575e9d99f","Type":"ContainerStarted","Data":"e0fd01a5bf2bf9f87759aa5a09adc73b2cb077c76f82813045d39459a79ccd69"} Jun 11 20:29:56.940717 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:56.940641 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:29:56.964461 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:56.964361 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" podStartSLOduration=1.491289625 podStartE2EDuration="4.964337613s" totalImagesPullingTime="3.473047988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:56.962019722 +0000 UTC m=+262.316766033" watchObservedRunningTime="2026-06-11 20:29:56.964337613 +0000 UTC m=+262.319083926" Jun 11 20:29:57.946067 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:57.946020 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" event={"ID":"24a55637-1ee5-4825-9a7b-2cae9e594ac5","Type":"ContainerStarted","Data":"f8dc56dc4c4a3041303354f4d3c6eb365cea023f35cb243228848879ea7ba77a"} Jun 11 20:29:57.964139 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:57.964074 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" podStartSLOduration=2.104863496 podStartE2EDuration="3.964054772s" totalImagesPullingTime="1.859191276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:57.961929022 +0000 UTC m=+263.316675341" watchObservedRunningTime="2026-06-11 20:29:57.964054772 +0000 UTC m=+263.318801087" Jun 11 20:29:58.951187 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:58.951142 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6954888546-vcghk" event={"ID":"43ad3810-fbf7-41b9-814a-7e4c1d5b812d","Type":"ContainerStarted","Data":"6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f"} Jun 11 20:29:58.969001 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:29:58.968877 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6954888546-vcghk" podStartSLOduration=1.027962495 podStartE2EDuration="3.968858354s" totalImagesPullingTime="2.940895859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:29:58.967626901 +0000 UTC m=+264.322373222" watchObservedRunningTime="2026-06-11 20:29:58.968858354 +0000 UTC m=+264.323604670" Jun 11 20:30:03.958104 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:03.958074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57c8cffbc7-8jq9m" Jun 11 20:30:05.641011 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:05.640974 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6954888546-vcghk" Jun 11 20:30:05.641400 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:05.641044 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6954888546-vcghk" Jun 11 20:30:05.642560 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:05.642535 2564 patch_prober.go:28] interesting pod/console-6954888546-vcghk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" start-of-body= Jun 11 20:30:05.642656 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:05.642591 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6954888546-vcghk" podUID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" containerName="console" probeResult="failure" output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" Jun 11 20:30:15.215623 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:15.215581 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:30:15.216050 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:15.215644 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:30:15.641495 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:15.641455 2564 patch_prober.go:28] interesting pod/console-6954888546-vcghk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" start-of-body= Jun 11 20:30:15.641663 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:15.641522 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6954888546-vcghk" podUID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" containerName="console" probeResult="failure" output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" Jun 11 20:30:18.027124 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:18.027080 2564 generic.go:356] "Generic (PLEG): container finished" podID="8bf7b5a5-bb18-47ff-909c-6c3201f88856" containerID="2bef83aeea795b18d723ed6f02c788623140e5d1934dea58fcc2e3f587c82672" exitCode=0 Jun 11 20:30:18.027536 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:18.027154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" event={"ID":"8bf7b5a5-bb18-47ff-909c-6c3201f88856","Type":"ContainerDied","Data":"2bef83aeea795b18d723ed6f02c788623140e5d1934dea58fcc2e3f587c82672"} Jun 11 20:30:18.027536 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:18.027500 2564 scope.go:117] "RemoveContainer" containerID="2bef83aeea795b18d723ed6f02c788623140e5d1934dea58fcc2e3f587c82672" Jun 11 20:30:18.574496 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:30:18.574439 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" podUID="a491ede1-e21e-4861-8023-2a5d7d331351" Jun 11 20:30:18.574496 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:30:18.574439 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qpm6t" podUID="e98f77d6-3aa7-4ace-8987-93b443679091" Jun 11 20:30:19.033217 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:19.033180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7ccbfd59dc-mf44h" event={"ID":"8bf7b5a5-bb18-47ff-909c-6c3201f88856","Type":"ContainerStarted","Data":"5e96ebb66f10e38f55faf68ef0b7f76068826e38bd62bfdbd13e29d387d5fec8"} Jun 11 20:30:19.033217 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:19.033219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpm6t" Jun 11 20:30:19.033757 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:19.033219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:30:22.016374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.016318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:30:22.016374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.016380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:30:22.018779 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.018753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e98f77d6-3aa7-4ace-8987-93b443679091-metrics-tls\") pod \"dns-default-qpm6t\" (UID: \"e98f77d6-3aa7-4ace-8987-93b443679091\") " pod="openshift-dns/dns-default-qpm6t" Jun 11 20:30:22.018889 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.018833 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"image-registry-6bbdc7f8b7-9qvxh\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:30:22.036090 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.036058 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jm4xk\"" Jun 11 20:30:22.036630 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.036614 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7qg4w\"" Jun 11 20:30:22.044412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.044368 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:30:22.044412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.044404 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpm6t" Jun 11 20:30:22.189761 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.189736 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh"] Jun 11 20:30:22.192699 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:30:22.192655 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda491ede1_e21e_4861_8023_2a5d7d331351.slice/crio-88bb72bd4ba4ad6a9569533561d9db7a635684f5b5fbc21d2a2fe218e8759c29 WatchSource:0}: Error finding container 88bb72bd4ba4ad6a9569533561d9db7a635684f5b5fbc21d2a2fe218e8759c29: Status 404 returned error can't find the container with id 88bb72bd4ba4ad6a9569533561d9db7a635684f5b5fbc21d2a2fe218e8759c29 Jun 11 20:30:22.203888 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:22.203865 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qpm6t"] Jun 11 20:30:22.207344 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:30:22.207311 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98f77d6_3aa7_4ace_8987_93b443679091.slice/crio-133fdb18f22a84a09380f10c53002738b99dcc2c8d71bc1aeec2350e5c077881 WatchSource:0}: Error finding container 133fdb18f22a84a09380f10c53002738b99dcc2c8d71bc1aeec2350e5c077881: Status 404 returned error can't find the container with id 133fdb18f22a84a09380f10c53002738b99dcc2c8d71bc1aeec2350e5c077881 Jun 11 20:30:23.051356 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:23.051314 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qpm6t" event={"ID":"e98f77d6-3aa7-4ace-8987-93b443679091","Type":"ContainerStarted","Data":"133fdb18f22a84a09380f10c53002738b99dcc2c8d71bc1aeec2350e5c077881"} Jun 11 20:30:23.053124 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:23.053095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" event={"ID":"a491ede1-e21e-4861-8023-2a5d7d331351","Type":"ContainerStarted","Data":"1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f"} Jun 11 20:30:23.053277 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:23.053144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" event={"ID":"a491ede1-e21e-4861-8023-2a5d7d331351","Type":"ContainerStarted","Data":"88bb72bd4ba4ad6a9569533561d9db7a635684f5b5fbc21d2a2fe218e8759c29"} Jun 11 20:30:23.053334 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:23.053303 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:30:23.071834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:23.071771 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" podStartSLOduration=288.071748406 podStartE2EDuration="4m48.071748406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:30:23.07072655 +0000 UTC m=+288.425472869" watchObservedRunningTime="2026-06-11 20:30:23.071748406 +0000 UTC m=+288.426494721" Jun 11 20:30:25.064932 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:25.064895 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qpm6t" event={"ID":"e98f77d6-3aa7-4ace-8987-93b443679091","Type":"ContainerStarted","Data":"44d5f14ad4dd1ed2c6fd6aa9d884e54886f2f050f0afb42b46d71971434991a6"} Jun 11 20:30:25.065437 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:25.065143 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qpm6t" Jun 11 20:30:25.065437 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:25.065184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qpm6t" event={"ID":"e98f77d6-3aa7-4ace-8987-93b443679091","Type":"ContainerStarted","Data":"c64943c4a90fc39d16da666dc97e8ce1ba8f2df5c2cc8119033d3c3abadb1a0e"} Jun 11 20:30:25.080433 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:25.080364 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qpm6t" podStartSLOduration=251.113880373 podStartE2EDuration="4m13.080345027s" totalImagesPullingTime="1.966464654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:30:25.079585123 +0000 UTC m=+290.434331452" watchObservedRunningTime="2026-06-11 20:30:25.080345027 +0000 UTC m=+290.435091344" Jun 11 20:30:25.641774 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:25.641732 2564 patch_prober.go:28] interesting pod/console-6954888546-vcghk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" start-of-body= Jun 11 20:30:25.641951 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:25.641791 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6954888546-vcghk" podUID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" containerName="console" probeResult="failure" output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" Jun 11 20:30:28.079062 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:28.079026 2564 generic.go:356] "Generic (PLEG): container finished" podID="98700412-c4d4-4989-8edf-90b8e43cf90d" containerID="79e561f8ff4bb78a9a240da18fac2ba1526bfdf5dccad9ee5bffbfdf09b20d2a" exitCode=0 Jun 11 20:30:28.079438 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:28.079104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" event={"ID":"98700412-c4d4-4989-8edf-90b8e43cf90d","Type":"ContainerDied","Data":"79e561f8ff4bb78a9a240da18fac2ba1526bfdf5dccad9ee5bffbfdf09b20d2a"} Jun 11 20:30:28.079438 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:28.079417 2564 scope.go:117] "RemoveContainer" containerID="79e561f8ff4bb78a9a240da18fac2ba1526bfdf5dccad9ee5bffbfdf09b20d2a" Jun 11 20:30:29.084916 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:29.084880 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7c578b78fc-lxq24" event={"ID":"98700412-c4d4-4989-8edf-90b8e43cf90d","Type":"ContainerStarted","Data":"535c83feca3ea1a7b793d6007c0d01e4d3bd17b778548624a82a1cfb5d958b3f"} Jun 11 20:30:33.101256 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:33.101222 2564 generic.go:356] "Generic (PLEG): container finished" podID="c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f" containerID="bac2e02f0e223b8ee61caa4d396ccb4e7b5831786a774541c5f620e502d5b872" exitCode=0 Jun 11 20:30:33.101641 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:33.101304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" event={"ID":"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f","Type":"ContainerDied","Data":"bac2e02f0e223b8ee61caa4d396ccb4e7b5831786a774541c5f620e502d5b872"} Jun 11 20:30:33.101682 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:33.101650 2564 scope.go:117] "RemoveContainer" containerID="bac2e02f0e223b8ee61caa4d396ccb4e7b5831786a774541c5f620e502d5b872" Jun 11 20:30:34.107012 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:34.106952 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5bbd86d6bd-z825d" event={"ID":"c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f","Type":"ContainerStarted","Data":"a8860b31842d31829d90ac027f65f52f65434d421ebcdb84be1d94a1bce5bdcd"} Jun 11 20:30:35.026573 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.026541 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:30:35.026804 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.026789 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:30:35.033803 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.033780 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:30:35.034157 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.034135 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:30:35.039386 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.039365 2564 kubelet.go:1648] "Image garbage collection succeeded" Jun 11 20:30:35.075338 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.075185 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qpm6t" Jun 11 20:30:35.221635 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.221606 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:30:35.225581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.225556 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84b49b6d67-9dvrm" Jun 11 20:30:35.641203 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.641166 2564 patch_prober.go:28] interesting pod/console-6954888546-vcghk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" start-of-body= Jun 11 20:30:35.641380 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:35.641217 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6954888546-vcghk" podUID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" containerName="console" probeResult="failure" output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" Jun 11 20:30:38.337374 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:38.337313 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84b49b6d67-9dvrm_24a55637-1ee5-4825-9a7b-2cae9e594ac5/metrics-server/0.log" Jun 11 20:30:39.337164 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:39.337133 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkqtr_933a7689-fe60-4573-ba8c-ec03a820295f/init-textfile/0.log" Jun 11 20:30:39.538670 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:39.538637 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkqtr_933a7689-fe60-4573-ba8c-ec03a820295f/node-exporter/0.log" Jun 11 20:30:39.736900 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:39.736821 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkqtr_933a7689-fe60-4573-ba8c-ec03a820295f/kube-rbac-proxy/0.log" Jun 11 20:30:40.542902 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:40.542871 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-bjkkp_c70e158e-548b-44a4-969d-d701a609aa7f/kube-rbac-proxy-main/0.log" Jun 11 20:30:40.736267 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:40.736233 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-bjkkp_c70e158e-548b-44a4-969d-d701a609aa7f/kube-rbac-proxy-self/0.log" Jun 11 20:30:40.936880 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:40.936797 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-bjkkp_c70e158e-548b-44a4-969d-d701a609aa7f/openshift-state-metrics/0.log" Jun 11 20:30:42.049461 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:42.049421 2564 patch_prober.go:28] interesting pod/image-registry-6bbdc7f8b7-9qvxh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 11 20:30:42.049825 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:42.049477 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" podUID="a491ede1-e21e-4861-8023-2a5d7d331351" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 11 20:30:42.936368 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:42.936330 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5cf667749c-krq5h_5f55f499-98f4-46dd-b574-9dbaf61c8c3a/prometheus-operator-admission-webhook/0.log" Jun 11 20:30:43.136417 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:43.136387 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/thanos-query/0.log" Jun 11 20:30:43.335941 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:43.335912 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/kube-rbac-proxy-web/0.log" Jun 11 20:30:43.536550 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:43.536522 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/kube-rbac-proxy/0.log" Jun 11 20:30:43.735753 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:43.735677 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/prom-label-proxy/0.log" Jun 11 20:30:43.936553 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:43.936519 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/kube-rbac-proxy-rules/0.log" Jun 11 20:30:44.063127 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:44.063097 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:30:44.135906 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:44.135873 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/kube-rbac-proxy-metrics/0.log" Jun 11 20:30:44.336528 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:44.336444 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-867b76cb74-dt22k_8a2c57c0-8141-4960-8b86-d8783a208d7e/networking-console-plugin/0.log" Jun 11 20:30:44.536920 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:44.536884 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:30:44.739736 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:44.739650 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/2.log" Jun 11 20:30:45.137176 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:45.137148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6954888546-vcghk_43ad3810-fbf7-41b9-814a-7e4c1d5b812d/console/0.log" Jun 11 20:30:45.645169 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:45.645138 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6954888546-vcghk" Jun 11 20:30:45.649213 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:45.649189 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6954888546-vcghk" Jun 11 20:30:46.536593 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:46.536565 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qpm6t_e98f77d6-3aa7-4ace-8987-93b443679091/dns/0.log" Jun 11 20:30:46.736726 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:46.736699 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qpm6t_e98f77d6-3aa7-4ace-8987-93b443679091/kube-rbac-proxy/0.log" Jun 11 20:30:46.936756 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:46.936674 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5fnqb_f748b10a-2034-4330-9292-f4781f1b4359/dns-node-resolver/0.log" Jun 11 20:30:47.736878 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:47.736833 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bbdc7f8b7-9qvxh_a491ede1-e21e-4861-8023-2a5d7d331351/registry/0.log" Jun 11 20:30:48.136073 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:48.136040 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k58bv_c0e098fe-1a43-4895-a7e0-ab36688d5b77/node-ca/0.log" Jun 11 20:30:48.536446 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:48.536413 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76bc48fcc4-kpmwf_fb12d4fb-c613-4500-9bf5-f09a5f001dc4/router/0.log" Jun 11 20:30:49.136253 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:49.136221 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fgbx2_01568624-0240-48ae-9f43-2bc8eb597dcd/serve-healthcheck-canary/0.log" Jun 11 20:30:56.193205 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:30:56.193168 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh"] Jun 11 20:31:20.338512 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.338476 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79dd95b74c-wqmhk"] Jun 11 20:31:20.344194 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.344165 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.353170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.353148 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79dd95b74c-wqmhk"] Jun 11 20:31:20.424088 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.424043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-serving-cert\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.424262 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.424109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-trusted-ca-bundle\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.424262 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.424128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-console-config\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.424262 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.424152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-oauth-config\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.424262 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.424169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-service-ca\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.424262 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.424240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mtg\" (UniqueName: \"kubernetes.io/projected/3049d2bd-4a96-4997-aa1e-e359a3424382-kube-api-access-g9mtg\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.424426 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.424278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-oauth-serving-cert\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.525378 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.525334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mtg\" (UniqueName: \"kubernetes.io/projected/3049d2bd-4a96-4997-aa1e-e359a3424382-kube-api-access-g9mtg\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.525378 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.525380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-oauth-serving-cert\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.525589 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.525501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-serving-cert\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.525632 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.525605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-trusted-ca-bundle\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.525677 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.525637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-console-config\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.525677 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.525672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-oauth-config\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.525753 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.525701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-service-ca\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.526141 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.526117 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-oauth-serving-cert\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.526433 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.526404 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-console-config\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.526529 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.526409 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-trusted-ca-bundle\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.526529 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.526450 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-service-ca\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.528170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.528147 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-serving-cert\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.528254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.528147 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-oauth-config\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.532902 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.532883 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mtg\" (UniqueName: \"kubernetes.io/projected/3049d2bd-4a96-4997-aa1e-e359a3424382-kube-api-access-g9mtg\") pod \"console-79dd95b74c-wqmhk\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.655622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.655532 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:20.786675 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:20.786649 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79dd95b74c-wqmhk"] Jun 11 20:31:20.788957 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:31:20.788927 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3049d2bd_4a96_4997_aa1e_e359a3424382.slice/crio-f9c3bcc7f3dacbc554da3096bde90de5e8f3d11fb00fb8e4feaa8ad9d18e0788 WatchSource:0}: Error finding container f9c3bcc7f3dacbc554da3096bde90de5e8f3d11fb00fb8e4feaa8ad9d18e0788: Status 404 returned error can't find the container with id f9c3bcc7f3dacbc554da3096bde90de5e8f3d11fb00fb8e4feaa8ad9d18e0788 Jun 11 20:31:21.215325 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.215262 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" podUID="a491ede1-e21e-4861-8023-2a5d7d331351" containerName="registry" containerID="cri-o://1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f" gracePeriod=30 Jun 11 20:31:21.299447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.299408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79dd95b74c-wqmhk" event={"ID":"3049d2bd-4a96-4997-aa1e-e359a3424382","Type":"ContainerStarted","Data":"19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948"} Jun 11 20:31:21.299614 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.299468 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79dd95b74c-wqmhk" event={"ID":"3049d2bd-4a96-4997-aa1e-e359a3424382","Type":"ContainerStarted","Data":"f9c3bcc7f3dacbc554da3096bde90de5e8f3d11fb00fb8e4feaa8ad9d18e0788"} Jun 11 20:31:21.316602 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.316544 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79dd95b74c-wqmhk" podStartSLOduration=1.316526407 podStartE2EDuration="1.316526407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:31:21.315687326 +0000 UTC m=+346.670433664" watchObservedRunningTime="2026-06-11 20:31:21.316526407 +0000 UTC m=+346.671272793" Jun 11 20:31:21.458115 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.458076 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:31:21.533795 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.533633 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-registry-certificates\") pod \"a491ede1-e21e-4861-8023-2a5d7d331351\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " Jun 11 20:31:21.533795 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.533682 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-installation-pull-secrets\") pod \"a491ede1-e21e-4861-8023-2a5d7d331351\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " Jun 11 20:31:21.533795 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.533719 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-image-registry-private-configuration\") pod \"a491ede1-e21e-4861-8023-2a5d7d331351\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " Jun 11 20:31:21.534072 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.533881 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-bound-sa-token\") pod \"a491ede1-e21e-4861-8023-2a5d7d331351\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " Jun 11 20:31:21.534072 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.533932 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a491ede1-e21e-4861-8023-2a5d7d331351-ca-trust-extracted\") pod \"a491ede1-e21e-4861-8023-2a5d7d331351\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " Jun 11 20:31:21.534072 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.533956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") pod \"a491ede1-e21e-4861-8023-2a5d7d331351\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " Jun 11 20:31:21.534072 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.534042 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx996\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-kube-api-access-cx996\") pod \"a491ede1-e21e-4861-8023-2a5d7d331351\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " Jun 11 20:31:21.534370 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.534099 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-trusted-ca\") pod \"a491ede1-e21e-4861-8023-2a5d7d331351\" (UID: \"a491ede1-e21e-4861-8023-2a5d7d331351\") " Jun 11 20:31:21.534370 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.534244 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a491ede1-e21e-4861-8023-2a5d7d331351" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:31:21.535089 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.535002 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a491ede1-e21e-4861-8023-2a5d7d331351" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:31:21.535610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.535559 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-trusted-ca\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:21.535610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.535584 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a491ede1-e21e-4861-8023-2a5d7d331351-registry-certificates\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:21.536868 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.536829 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-kube-api-access-cx996" (OuterVolumeSpecName: "kube-api-access-cx996") pod "a491ede1-e21e-4861-8023-2a5d7d331351" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351"). InnerVolumeSpecName "kube-api-access-cx996". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:31:21.537291 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.537255 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a491ede1-e21e-4861-8023-2a5d7d331351" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:31:21.537396 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.537281 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a491ede1-e21e-4861-8023-2a5d7d331351" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:31:21.537951 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.537913 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a491ede1-e21e-4861-8023-2a5d7d331351" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:31:21.538083 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.538049 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a491ede1-e21e-4861-8023-2a5d7d331351" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:31:21.543072 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.543044 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a491ede1-e21e-4861-8023-2a5d7d331351-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a491ede1-e21e-4861-8023-2a5d7d331351" (UID: "a491ede1-e21e-4861-8023-2a5d7d331351"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:31:21.636880 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.636843 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-image-registry-private-configuration\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:21.636880 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.636874 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-bound-sa-token\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:21.636880 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.636884 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a491ede1-e21e-4861-8023-2a5d7d331351-ca-trust-extracted\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:21.636880 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.636892 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:21.637165 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.636901 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx996\" (UniqueName: \"kubernetes.io/projected/a491ede1-e21e-4861-8023-2a5d7d331351-kube-api-access-cx996\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:21.637165 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:21.636911 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a491ede1-e21e-4861-8023-2a5d7d331351-installation-pull-secrets\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:22.304155 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:22.304120 2564 generic.go:356] "Generic (PLEG): container finished" podID="a491ede1-e21e-4861-8023-2a5d7d331351" containerID="1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f" exitCode=0 Jun 11 20:31:22.304337 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:22.304165 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" Jun 11 20:31:22.304337 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:22.304211 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" event={"ID":"a491ede1-e21e-4861-8023-2a5d7d331351","Type":"ContainerDied","Data":"1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f"} Jun 11 20:31:22.304337 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:22.304263 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh" event={"ID":"a491ede1-e21e-4861-8023-2a5d7d331351","Type":"ContainerDied","Data":"88bb72bd4ba4ad6a9569533561d9db7a635684f5b5fbc21d2a2fe218e8759c29"} Jun 11 20:31:22.304337 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:22.304291 2564 scope.go:117] "RemoveContainer" containerID="1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f" Jun 11 20:31:22.312953 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:22.312936 2564 scope.go:117] "RemoveContainer" containerID="1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f" Jun 11 20:31:22.313233 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:31:22.313216 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f\": container with ID starting with 1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f not found: ID does not exist" containerID="1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f" Jun 11 20:31:22.313290 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:22.313243 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f"} err="failed to get container status \"1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f\": rpc error: code = NotFound desc = could not find container \"1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f\": container with ID starting with 1aaf033df206ef8368cb6e80b8569d635b86e0929cdba0630fdc052bd8c41e7f not found: ID does not exist" Jun 11 20:31:22.324193 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:22.324167 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh"] Jun 11 20:31:22.327864 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:22.327841 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bbdc7f8b7-9qvxh"] Jun 11 20:31:23.075102 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:23.075064 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a491ede1-e21e-4861-8023-2a5d7d331351" path="/var/lib/kubelet/pods/a491ede1-e21e-4861-8023-2a5d7d331351/volumes" Jun 11 20:31:30.656171 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:30.656114 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:30.656574 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:30.656208 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:30.661031 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:30.661008 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:31.346642 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:31.346614 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:31:31.389889 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:31.389854 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6954888546-vcghk"] Jun 11 20:31:56.412436 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.412392 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6954888546-vcghk" podUID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" containerName="console" containerID="cri-o://6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f" gracePeriod=15 Jun 11 20:31:56.679416 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.679385 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6954888546-vcghk" Jun 11 20:31:56.739512 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.739479 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-oauth-config\") pod \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " Jun 11 20:31:56.739692 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.739529 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-oauth-serving-cert\") pod \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " Jun 11 20:31:56.739692 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.739546 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-serving-cert\") pod \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " Jun 11 20:31:56.739692 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.739564 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-service-ca\") pod \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " Jun 11 20:31:56.739692 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.739583 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-trusted-ca-bundle\") pod \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " Jun 11 20:31:56.739875 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.739772 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbgvd\" (UniqueName: \"kubernetes.io/projected/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-kube-api-access-tbgvd\") pod \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " Jun 11 20:31:56.739875 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.739812 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-config\") pod \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\" (UID: \"43ad3810-fbf7-41b9-814a-7e4c1d5b812d\") " Jun 11 20:31:56.740065 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.740010 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-service-ca" (OuterVolumeSpecName: "service-ca") pod "43ad3810-fbf7-41b9-814a-7e4c1d5b812d" (UID: "43ad3810-fbf7-41b9-814a-7e4c1d5b812d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:31:56.740065 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.740024 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "43ad3810-fbf7-41b9-814a-7e4c1d5b812d" (UID: "43ad3810-fbf7-41b9-814a-7e4c1d5b812d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:31:56.740065 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.740038 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "43ad3810-fbf7-41b9-814a-7e4c1d5b812d" (UID: "43ad3810-fbf7-41b9-814a-7e4c1d5b812d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:31:56.740280 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.740112 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-oauth-serving-cert\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:56.740280 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.740130 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-service-ca\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:56.740280 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.740142 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-trusted-ca-bundle\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:56.740430 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.740278 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-config" (OuterVolumeSpecName: "console-config") pod "43ad3810-fbf7-41b9-814a-7e4c1d5b812d" (UID: "43ad3810-fbf7-41b9-814a-7e4c1d5b812d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:31:56.741847 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.741820 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "43ad3810-fbf7-41b9-814a-7e4c1d5b812d" (UID: "43ad3810-fbf7-41b9-814a-7e4c1d5b812d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:31:56.741952 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.741884 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-kube-api-access-tbgvd" (OuterVolumeSpecName: "kube-api-access-tbgvd") pod "43ad3810-fbf7-41b9-814a-7e4c1d5b812d" (UID: "43ad3810-fbf7-41b9-814a-7e4c1d5b812d"). InnerVolumeSpecName "kube-api-access-tbgvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:31:56.741952 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.741891 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "43ad3810-fbf7-41b9-814a-7e4c1d5b812d" (UID: "43ad3810-fbf7-41b9-814a-7e4c1d5b812d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:31:56.841309 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.841271 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-oauth-config\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:56.841309 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.841301 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-serving-cert\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:56.841309 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.841311 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbgvd\" (UniqueName: \"kubernetes.io/projected/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-kube-api-access-tbgvd\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:56.841309 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:56.841320 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43ad3810-fbf7-41b9-814a-7e4c1d5b812d-console-config\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:31:57.452993 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:57.452883 2564 generic.go:356] "Generic (PLEG): container finished" podID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" containerID="6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f" exitCode=0 Jun 11 20:31:57.452993 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:57.452933 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6954888546-vcghk" Jun 11 20:31:57.453449 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:57.452986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6954888546-vcghk" event={"ID":"43ad3810-fbf7-41b9-814a-7e4c1d5b812d","Type":"ContainerDied","Data":"6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f"} Jun 11 20:31:57.453449 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:57.453038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6954888546-vcghk" event={"ID":"43ad3810-fbf7-41b9-814a-7e4c1d5b812d","Type":"ContainerDied","Data":"524a5fbd86aba709940c1450cf1fd334fb1e3f677a10b21a30074aa70de850ac"} Jun 11 20:31:57.453449 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:57.453062 2564 scope.go:117] "RemoveContainer" containerID="6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f" Jun 11 20:31:57.461111 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:57.461091 2564 scope.go:117] "RemoveContainer" containerID="6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f" Jun 11 20:31:57.461418 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:31:57.461398 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f\": container with ID starting with 6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f not found: ID does not exist" containerID="6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f" Jun 11 20:31:57.461502 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:57.461424 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f"} err="failed to get container status \"6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f\": rpc error: code = NotFound desc = could not find container \"6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f\": container with ID starting with 6d75bf4112e3096e768786c77a567c6745c0854224e5a0263c4815a666ffbf1f not found: ID does not exist" Jun 11 20:31:57.468618 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:57.468592 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6954888546-vcghk"] Jun 11 20:31:57.475703 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:57.475675 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6954888546-vcghk"] Jun 11 20:31:59.075081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:31:59.075045 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" path="/var/lib/kubelet/pods/43ad3810-fbf7-41b9-814a-7e4c1d5b812d/volumes" Jun 11 20:32:39.616091 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:32:39.616053 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79dd95b74c-wqmhk"] Jun 11 20:33:04.639974 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.639853 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-79dd95b74c-wqmhk" podUID="3049d2bd-4a96-4997-aa1e-e359a3424382" containerName="console" containerID="cri-o://19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948" gracePeriod=15 Jun 11 20:33:04.895564 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.895505 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:33:04.993519 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.993480 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-oauth-config\") pod \"3049d2bd-4a96-4997-aa1e-e359a3424382\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " Jun 11 20:33:04.993708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.993543 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-serving-cert\") pod \"3049d2bd-4a96-4997-aa1e-e359a3424382\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " Jun 11 20:33:04.993708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.993577 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9mtg\" (UniqueName: \"kubernetes.io/projected/3049d2bd-4a96-4997-aa1e-e359a3424382-kube-api-access-g9mtg\") pod \"3049d2bd-4a96-4997-aa1e-e359a3424382\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " Jun 11 20:33:04.993708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.993612 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-console-config\") pod \"3049d2bd-4a96-4997-aa1e-e359a3424382\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " Jun 11 20:33:04.993708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.993654 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-trusted-ca-bundle\") pod \"3049d2bd-4a96-4997-aa1e-e359a3424382\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " Jun 11 20:33:04.993708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.993682 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-oauth-serving-cert\") pod \"3049d2bd-4a96-4997-aa1e-e359a3424382\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " Jun 11 20:33:04.994006 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.993712 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-service-ca\") pod \"3049d2bd-4a96-4997-aa1e-e359a3424382\" (UID: \"3049d2bd-4a96-4997-aa1e-e359a3424382\") " Jun 11 20:33:04.994196 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.994163 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-console-config" (OuterVolumeSpecName: "console-config") pod "3049d2bd-4a96-4997-aa1e-e359a3424382" (UID: "3049d2bd-4a96-4997-aa1e-e359a3424382"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:33:04.994196 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.994181 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3049d2bd-4a96-4997-aa1e-e359a3424382" (UID: "3049d2bd-4a96-4997-aa1e-e359a3424382"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:33:04.994343 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.994211 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3049d2bd-4a96-4997-aa1e-e359a3424382" (UID: "3049d2bd-4a96-4997-aa1e-e359a3424382"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:33:04.994343 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.994280 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-service-ca" (OuterVolumeSpecName: "service-ca") pod "3049d2bd-4a96-4997-aa1e-e359a3424382" (UID: "3049d2bd-4a96-4997-aa1e-e359a3424382"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:33:04.995808 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.995780 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3049d2bd-4a96-4997-aa1e-e359a3424382" (UID: "3049d2bd-4a96-4997-aa1e-e359a3424382"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:33:04.995808 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.995798 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3049d2bd-4a96-4997-aa1e-e359a3424382-kube-api-access-g9mtg" (OuterVolumeSpecName: "kube-api-access-g9mtg") pod "3049d2bd-4a96-4997-aa1e-e359a3424382" (UID: "3049d2bd-4a96-4997-aa1e-e359a3424382"). InnerVolumeSpecName "kube-api-access-g9mtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:33:04.995938 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:04.995875 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3049d2bd-4a96-4997-aa1e-e359a3424382" (UID: "3049d2bd-4a96-4997-aa1e-e359a3424382"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:33:05.094269 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.094236 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-serving-cert\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:33:05.094269 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.094265 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9mtg\" (UniqueName: \"kubernetes.io/projected/3049d2bd-4a96-4997-aa1e-e359a3424382-kube-api-access-g9mtg\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:33:05.094269 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.094276 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-console-config\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:33:05.094492 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.094286 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-trusted-ca-bundle\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:33:05.094492 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.094295 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-oauth-serving-cert\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:33:05.094492 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.094303 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3049d2bd-4a96-4997-aa1e-e359a3424382-service-ca\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:33:05.094492 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.094311 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3049d2bd-4a96-4997-aa1e-e359a3424382-console-oauth-config\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:33:05.719333 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.719295 2564 generic.go:356] "Generic (PLEG): container finished" podID="3049d2bd-4a96-4997-aa1e-e359a3424382" containerID="19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948" exitCode=0 Jun 11 20:33:05.719796 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.719357 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79dd95b74c-wqmhk" Jun 11 20:33:05.719796 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.719377 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79dd95b74c-wqmhk" event={"ID":"3049d2bd-4a96-4997-aa1e-e359a3424382","Type":"ContainerDied","Data":"19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948"} Jun 11 20:33:05.719796 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.719422 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79dd95b74c-wqmhk" event={"ID":"3049d2bd-4a96-4997-aa1e-e359a3424382","Type":"ContainerDied","Data":"f9c3bcc7f3dacbc554da3096bde90de5e8f3d11fb00fb8e4feaa8ad9d18e0788"} Jun 11 20:33:05.719796 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.719443 2564 scope.go:117] "RemoveContainer" containerID="19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948" Jun 11 20:33:05.727644 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.727627 2564 scope.go:117] "RemoveContainer" containerID="19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948" Jun 11 20:33:05.727915 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:33:05.727890 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948\": container with ID starting with 19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948 not found: ID does not exist" containerID="19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948" Jun 11 20:33:05.728033 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.727924 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948"} err="failed to get container status \"19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948\": rpc error: code = NotFound desc = could not find container \"19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948\": container with ID starting with 19720dab0856b61888223c54d7b658c4b5f141f224cd9b50f46f49e3b1708948 not found: ID does not exist" Jun 11 20:33:05.735586 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.735558 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79dd95b74c-wqmhk"] Jun 11 20:33:05.741440 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:05.741416 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79dd95b74c-wqmhk"] Jun 11 20:33:07.074775 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:33:07.074741 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3049d2bd-4a96-4997-aa1e-e359a3424382" path="/var/lib/kubelet/pods/3049d2bd-4a96-4997-aa1e-e359a3424382/volumes" Jun 11 20:34:59.742034 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742000 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4"] Jun 11 20:34:59.742526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742334 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" containerName="console" Jun 11 20:34:59.742526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742345 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" containerName="console" Jun 11 20:34:59.742526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742360 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a491ede1-e21e-4861-8023-2a5d7d331351" containerName="registry" Jun 11 20:34:59.742526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742365 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a491ede1-e21e-4861-8023-2a5d7d331351" containerName="registry" Jun 11 20:34:59.742526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742376 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3049d2bd-4a96-4997-aa1e-e359a3424382" containerName="console" Jun 11 20:34:59.742526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742382 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3049d2bd-4a96-4997-aa1e-e359a3424382" containerName="console" Jun 11 20:34:59.742526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742433 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a491ede1-e21e-4861-8023-2a5d7d331351" containerName="registry" Jun 11 20:34:59.742526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742440 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3049d2bd-4a96-4997-aa1e-e359a3424382" containerName="console" Jun 11 20:34:59.742526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.742446 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ad3810-fbf7-41b9-814a-7e4c1d5b812d" containerName="console" Jun 11 20:34:59.745185 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.745170 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" Jun 11 20:34:59.747283 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.747258 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jun 11 20:34:59.748075 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.748047 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jun 11 20:34:59.748184 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.748055 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-xnnr4\"" Jun 11 20:34:59.759350 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.759318 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4"] Jun 11 20:34:59.879934 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.879888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96kx\" (UniqueName: \"kubernetes.io/projected/df5524a3-60eb-413e-bc6e-4d50f3b14744-kube-api-access-f96kx\") pod \"feast-operator-controller-manager-5b4f6ff995-ck9f4\" (UID: \"df5524a3-60eb-413e-bc6e-4d50f3b14744\") " pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" Jun 11 20:34:59.980633 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.980590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96kx\" (UniqueName: \"kubernetes.io/projected/df5524a3-60eb-413e-bc6e-4d50f3b14744-kube-api-access-f96kx\") pod \"feast-operator-controller-manager-5b4f6ff995-ck9f4\" (UID: \"df5524a3-60eb-413e-bc6e-4d50f3b14744\") " pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" Jun 11 20:34:59.988491 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:34:59.988451 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96kx\" (UniqueName: \"kubernetes.io/projected/df5524a3-60eb-413e-bc6e-4d50f3b14744-kube-api-access-f96kx\") pod \"feast-operator-controller-manager-5b4f6ff995-ck9f4\" (UID: \"df5524a3-60eb-413e-bc6e-4d50f3b14744\") " pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" Jun 11 20:35:00.055165 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:00.055129 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" Jun 11 20:35:00.185453 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:00.185421 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4"] Jun 11 20:35:00.188390 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:35:00.188360 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5524a3_60eb_413e_bc6e_4d50f3b14744.slice/crio-d78b2048d3bf4e1a59ae6d2c99f4aa252a61a178626522b5fe5f9a5cef6bb412 WatchSource:0}: Error finding container d78b2048d3bf4e1a59ae6d2c99f4aa252a61a178626522b5fe5f9a5cef6bb412: Status 404 returned error can't find the container with id d78b2048d3bf4e1a59ae6d2c99f4aa252a61a178626522b5fe5f9a5cef6bb412 Jun 11 20:35:00.190427 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:00.190409 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 11 20:35:01.178193 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:01.178151 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" event={"ID":"df5524a3-60eb-413e-bc6e-4d50f3b14744","Type":"ContainerStarted","Data":"d78b2048d3bf4e1a59ae6d2c99f4aa252a61a178626522b5fe5f9a5cef6bb412"} Jun 11 20:35:04.192597 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:04.192562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" event={"ID":"df5524a3-60eb-413e-bc6e-4d50f3b14744","Type":"ContainerStarted","Data":"7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626"} Jun 11 20:35:04.193000 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:04.192646 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" Jun 11 20:35:04.207901 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:04.207842 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" podStartSLOduration=1.837681041 podStartE2EDuration="5.207825452s" totalImagesPullingTime="3.370144411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:35:04.206242185 +0000 UTC m=+569.560988514" watchObservedRunningTime="2026-06-11 20:35:04.207825452 +0000 UTC m=+569.562571767" Jun 11 20:35:15.198167 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:15.198138 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" Jun 11 20:35:35.054070 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:35.054041 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:35:35.054754 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:35.054735 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:35:35.060661 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:35.060631 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:35:35.061150 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:35.061134 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:35:43.478238 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.478196 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h"] Jun 11 20:35:43.486309 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.486277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.488582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.488553 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 11 20:35:43.488716 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.488586 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jun 11 20:35:43.488858 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.488840 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 11 20:35:43.488931 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.488908 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 11 20:35:43.489075 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.489049 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 11 20:35:43.489225 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.489209 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 11 20:35:43.489333 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.489316 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-ws7l8\"" Jun 11 20:35:43.489414 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.489400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 11 20:35:43.493795 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.493766 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h"] Jun 11 20:35:43.543022 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.542989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qbv\" (UniqueName: \"kubernetes.io/projected/4666c749-9601-462c-a180-cd61392e53a7-kube-api-access-r5qbv\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.543196 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.543082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-ui-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.543196 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.543106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4666c749-9601-462c-a180-cd61392e53a7-feast-data\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.543196 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.543129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-online-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.543305 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.543227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-offline-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.543339 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.543308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-registry-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.644262 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.644214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-ui-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.644262 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.644268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4666c749-9601-462c-a180-cd61392e53a7-feast-data\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.644536 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.644300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-online-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.644536 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.644327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-offline-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.644536 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.644378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-registry-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.644536 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:35:43.644464 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:35:43.644536 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:35:43.644471 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:35:43.644536 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:35:43.644539 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-registry-tls podName:4666c749-9601-462c-a180-cd61392e53a7 nodeName:}" failed. No retries permitted until 2026-06-11 20:35:44.1445159 +0000 UTC m=+609.499262200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-registry-tls") pod "feast-simple-feast-setup-846f56fdd8-7kn6h" (UID: "4666c749-9601-462c-a180-cd61392e53a7") : secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:35:43.644837 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:35:43.644556 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-offline-tls podName:4666c749-9601-462c-a180-cd61392e53a7 nodeName:}" failed. No retries permitted until 2026-06-11 20:35:44.144547482 +0000 UTC m=+609.499293779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-offline-tls") pod "feast-simple-feast-setup-846f56fdd8-7kn6h" (UID: "4666c749-9601-462c-a180-cd61392e53a7") : secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:35:43.644837 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.644572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qbv\" (UniqueName: \"kubernetes.io/projected/4666c749-9601-462c-a180-cd61392e53a7-kube-api-access-r5qbv\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.644936 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.644866 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4666c749-9601-462c-a180-cd61392e53a7-feast-data\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.647206 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.647165 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-online-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.647363 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.647236 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-ui-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:43.652685 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:43.652658 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qbv\" (UniqueName: \"kubernetes.io/projected/4666c749-9601-462c-a180-cd61392e53a7-kube-api-access-r5qbv\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:44.148469 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:44.148428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-registry-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:44.148677 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:44.148542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-offline-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:44.150909 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:44.150883 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-registry-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:44.151069 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:44.150918 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-offline-tls\") pod \"feast-simple-feast-setup-846f56fdd8-7kn6h\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:44.405898 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:44.405800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:35:44.546528 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:44.546500 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h"] Jun 11 20:35:44.548271 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:35:44.548242 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4666c749_9601_462c_a180_cd61392e53a7.slice/crio-2e6d6357230d1447859ed2b053e78aeebd02fd04489573af5eb77f037e152956 WatchSource:0}: Error finding container 2e6d6357230d1447859ed2b053e78aeebd02fd04489573af5eb77f037e152956: Status 404 returned error can't find the container with id 2e6d6357230d1447859ed2b053e78aeebd02fd04489573af5eb77f037e152956 Jun 11 20:35:45.357431 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:35:45.357391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerStarted","Data":"2e6d6357230d1447859ed2b053e78aeebd02fd04489573af5eb77f037e152956"} Jun 11 20:36:19.514762 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:19.514726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerStarted","Data":"712b4edd2431192296a1a05d72be59bdb4945cf0e94654398e143caaa1289c7e"} Jun 11 20:36:24.537804 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:24.537766 2564 generic.go:356] "Generic (PLEG): container finished" podID="4666c749-9601-462c-a180-cd61392e53a7" containerID="712b4edd2431192296a1a05d72be59bdb4945cf0e94654398e143caaa1289c7e" exitCode=0 Jun 11 20:36:24.538293 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:24.537846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerDied","Data":"712b4edd2431192296a1a05d72be59bdb4945cf0e94654398e143caaa1289c7e"} Jun 11 20:36:25.544641 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:25.544602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerStarted","Data":"07f8ed6f51310c8c314907bfa0efeb50cd152b2ff48c812ccb0830c3c3f3f89b"} Jun 11 20:36:30.566847 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:30.566760 2564 generic.go:356] "Generic (PLEG): container finished" podID="4666c749-9601-462c-a180-cd61392e53a7" containerID="07f8ed6f51310c8c314907bfa0efeb50cd152b2ff48c812ccb0830c3c3f3f89b" exitCode=0 Jun 11 20:36:30.566847 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:30.566800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerDied","Data":"07f8ed6f51310c8c314907bfa0efeb50cd152b2ff48c812ccb0830c3c3f3f89b"} Jun 11 20:36:31.576823 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:31.576785 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerStarted","Data":"42484db78b65abdee5cc855512e1876923ffc4c764137f1532e6707e3237cf92"} Jun 11 20:36:31.577401 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:31.577376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerStarted","Data":"d17f5ada8eef616b1f8cc8b3f9536188a52cbb29e9931cc1e1a2e3fa4ab19d4e"} Jun 11 20:36:31.577516 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:31.577501 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerStarted","Data":"d62db6f78ffcf93809798ef07cfd3422736705035fda68273448ae2698f351a2"} Jun 11 20:36:31.577638 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:31.577623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerStarted","Data":"f28b7e9dd507ea7bb0c2dbea9b4d2db31ba433148af7ed65521dbf96d66b1133"} Jun 11 20:36:31.599494 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:31.599431 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podStartSLOduration=14.151750398 podStartE2EDuration="48.599407944s" totalImagesPullingTime="34.447657546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:36:31.597598013 +0000 UTC m=+656.952344353" watchObservedRunningTime="2026-06-11 20:36:31.599407944 +0000 UTC m=+656.954154260" Jun 11 20:36:32.406297 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:32.406257 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:32.406587 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:32.406557 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:32.406587 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:32.406592 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:32.406780 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:32.406611 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:32.408904 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:32.408868 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.26:8816: connect: connection refused" Jun 11 20:36:32.409182 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:32.408868 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Jun 11 20:36:32.409306 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:32.408884 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.26:6571: connect: connection refused" Jun 11 20:36:32.409306 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:32.408920 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Jun 11 20:36:34.406117 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:34.406062 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:34.406614 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:34.406137 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:34.406614 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:34.406159 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:34.406614 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:34.406177 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:35.407085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:35.407034 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.26:8816: connect: connection refused" Jun 11 20:36:35.407601 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:35.407041 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Jun 11 20:36:35.407601 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:35.407040 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Jun 11 20:36:35.407936 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:35.407906 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.26:6571: connect: connection refused" Jun 11 20:36:38.407739 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:38.407703 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:38.408187 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:38.407984 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:38.408440 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:38.408424 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:38.408524 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:38.408495 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:38.408578 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:38.408545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:38.412366 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:38.412336 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:38.614434 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:38.614399 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:36:38.617091 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:36:38.617068 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:37:40.848626 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:40.848423 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h"] Jun 11 20:37:40.849219 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:40.848754 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="registry" containerID="cri-o://f28b7e9dd507ea7bb0c2dbea9b4d2db31ba433148af7ed65521dbf96d66b1133" gracePeriod=30 Jun 11 20:37:40.849219 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:40.849033 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="ui" containerID="cri-o://42484db78b65abdee5cc855512e1876923ffc4c764137f1532e6707e3237cf92" gracePeriod=30 Jun 11 20:37:40.849219 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:40.849158 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="online" containerID="cri-o://d62db6f78ffcf93809798ef07cfd3422736705035fda68273448ae2698f351a2" gracePeriod=30 Jun 11 20:37:40.849415 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:40.849328 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="offline" containerID="cri-o://d17f5ada8eef616b1f8cc8b3f9536188a52cbb29e9931cc1e1a2e3fa4ab19d4e" gracePeriod=30 Jun 11 20:37:41.099009 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.098904 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq"] Jun 11 20:37:41.103582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.103560 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.108170 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.108136 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-xz2j9\"" Jun 11 20:37:41.113461 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.113396 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq"] Jun 11 20:37:41.166095 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.166039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q22mz\" (UniqueName: \"kubernetes.io/projected/f0a49ccc-a500-4cba-a668-e87473dcfcd9-kube-api-access-q22mz\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.166095 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.166090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-offline-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.166369 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.166114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-ui-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.166369 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.166165 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-online-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.166369 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.166240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a49ccc-a500-4cba-a668-e87473dcfcd9-feast-data\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.166369 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.166351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-registry-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.267432 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.267384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-registry-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.267657 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.267451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q22mz\" (UniqueName: \"kubernetes.io/projected/f0a49ccc-a500-4cba-a668-e87473dcfcd9-kube-api-access-q22mz\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.267657 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.267486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-offline-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.267657 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.267511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-ui-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.267657 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.267558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-online-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.267657 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.267586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a49ccc-a500-4cba-a668-e87473dcfcd9-feast-data\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.267893 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:37:41.267557 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:37:41.267940 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:37:41.267932 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-registry-tls podName:f0a49ccc-a500-4cba-a668-e87473dcfcd9 nodeName:}" failed. No retries permitted until 2026-06-11 20:37:41.767909016 +0000 UTC m=+727.122655334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-registry-tls") pod "feast-simple-feast-setup-846f56fdd8-cg2wq" (UID: "f0a49ccc-a500-4cba-a668-e87473dcfcd9") : secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:37:41.268064 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:37:41.268048 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:37:41.268144 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:37:41.268130 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-ui-tls podName:f0a49ccc-a500-4cba-a668-e87473dcfcd9 nodeName:}" failed. No retries permitted until 2026-06-11 20:37:41.768114708 +0000 UTC m=+727.122861008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-ui-tls") pod "feast-simple-feast-setup-846f56fdd8-cg2wq" (UID: "f0a49ccc-a500-4cba-a668-e87473dcfcd9") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:37:41.268389 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.268353 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a49ccc-a500-4cba-a668-e87473dcfcd9-feast-data\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.270849 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.270822 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-offline-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.271304 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.271280 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-online-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.276403 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.276375 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q22mz\" (UniqueName: \"kubernetes.io/projected/f0a49ccc-a500-4cba-a668-e87473dcfcd9-kube-api-access-q22mz\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.772931 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.772883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-registry-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.772931 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.772942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-ui-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.775404 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.775370 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-ui-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.775404 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.775392 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-registry-tls\") pod \"feast-simple-feast-setup-846f56fdd8-cg2wq\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:41.870757 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.870720 2564 generic.go:356] "Generic (PLEG): container finished" podID="4666c749-9601-462c-a180-cd61392e53a7" containerID="42484db78b65abdee5cc855512e1876923ffc4c764137f1532e6707e3237cf92" exitCode=0 Jun 11 20:37:41.870757 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.870748 2564 generic.go:356] "Generic (PLEG): container finished" podID="4666c749-9601-462c-a180-cd61392e53a7" containerID="d62db6f78ffcf93809798ef07cfd3422736705035fda68273448ae2698f351a2" exitCode=0 Jun 11 20:37:41.871197 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.870790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerDied","Data":"42484db78b65abdee5cc855512e1876923ffc4c764137f1532e6707e3237cf92"} Jun 11 20:37:41.871197 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:41.870831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerDied","Data":"d62db6f78ffcf93809798ef07cfd3422736705035fda68273448ae2698f351a2"} Jun 11 20:37:42.016906 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:42.016864 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:42.158125 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:42.158094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq"] Jun 11 20:37:42.160195 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:37:42.160164 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0a49ccc_a500_4cba_a668_e87473dcfcd9.slice/crio-0d097b36e15bfb7b88c23dead856180e7917343982f8c486e5a2d984cce7a977 WatchSource:0}: Error finding container 0d097b36e15bfb7b88c23dead856180e7917343982f8c486e5a2d984cce7a977: Status 404 returned error can't find the container with id 0d097b36e15bfb7b88c23dead856180e7917343982f8c486e5a2d984cce7a977 Jun 11 20:37:42.877157 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:42.877119 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerStarted","Data":"f96a0cc9157ce736a9858bef613f095ef1f92aa4f8608c9759c6d3c14430f816"} Jun 11 20:37:42.877537 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:42.877174 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerStarted","Data":"0d097b36e15bfb7b88c23dead856180e7917343982f8c486e5a2d984cce7a977"} Jun 11 20:37:47.902324 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:47.902289 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerID="f96a0cc9157ce736a9858bef613f095ef1f92aa4f8608c9759c6d3c14430f816" exitCode=0 Jun 11 20:37:47.902851 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:47.902371 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerDied","Data":"f96a0cc9157ce736a9858bef613f095ef1f92aa4f8608c9759c6d3c14430f816"} Jun 11 20:37:48.408670 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:48.408631 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Jun 11 20:37:48.615190 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:48.615144 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Jun 11 20:37:48.908458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:48.908424 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerStarted","Data":"f663cfa0141ee0b99aa66a3d5550ab9782bd6df6fe4b875b5b474525c775c6c5"} Jun 11 20:37:53.932767 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:53.932680 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerID="f663cfa0141ee0b99aa66a3d5550ab9782bd6df6fe4b875b5b474525c775c6c5" exitCode=0 Jun 11 20:37:53.933158 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:53.932754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerDied","Data":"f663cfa0141ee0b99aa66a3d5550ab9782bd6df6fe4b875b5b474525c775c6c5"} Jun 11 20:37:54.942673 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:54.942630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerStarted","Data":"21330bc8ec958f598701878b8d23eff73a0364997a48f5192d6d06d685a9836d"} Jun 11 20:37:54.943182 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:54.942700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerStarted","Data":"efa8bd04b429e28e3c7dee468a63acc74ec4b8ad57a640fa27c03501f5e6708f"} Jun 11 20:37:54.943182 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:54.942720 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerStarted","Data":"adc03fabfdc23fd1cf56902ecff5f08f136ae49203da71ded9d3b07f56195703"} Jun 11 20:37:54.943182 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:54.942740 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerStarted","Data":"f305c357184aa05ddab0f549ea64abf7bf95f59f86673e96526775aa6fefe6c2"} Jun 11 20:37:54.966151 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:54.966078 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podStartSLOduration=13.966054104 podStartE2EDuration="13.966054104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:37:54.964664159 +0000 UTC m=+740.319410478" watchObservedRunningTime="2026-06-11 20:37:54.966054104 +0000 UTC m=+740.320800492" Jun 11 20:37:57.018015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:57.017854 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:57.018015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:57.017917 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:57.018015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:57.017939 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:57.018015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:57.017956 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:37:57.020250 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:57.020213 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.27:8816: connect: connection refused" Jun 11 20:37:57.020525 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:57.020211 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.27:6571: connect: connection refused" Jun 11 20:37:57.020604 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:57.020263 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Jun 11 20:37:57.020604 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:57.020323 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Jun 11 20:37:58.408493 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:58.408436 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Jun 11 20:37:58.615351 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:37:58.615286 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Jun 11 20:38:00.018327 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.018297 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.018872 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.018850 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.018917 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.018895 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.018986 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.018952 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.019099 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.019087 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.023118 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.023098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.976141 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.976102 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.976354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.976159 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.976354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.976177 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.976603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.976581 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.976720 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.976653 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:00.979088 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:00.979067 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:38:08.409110 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:08.409065 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" Jun 11 20:38:08.409598 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:08.409213 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:38:08.615075 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:08.615030 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" Jun 11 20:38:08.615238 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:08.615220 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:38:11.024529 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.024497 2564 generic.go:356] "Generic (PLEG): container finished" podID="4666c749-9601-462c-a180-cd61392e53a7" containerID="d17f5ada8eef616b1f8cc8b3f9536188a52cbb29e9931cc1e1a2e3fa4ab19d4e" exitCode=137 Jun 11 20:38:11.024529 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.024523 2564 generic.go:356] "Generic (PLEG): container finished" podID="4666c749-9601-462c-a180-cd61392e53a7" containerID="f28b7e9dd507ea7bb0c2dbea9b4d2db31ba433148af7ed65521dbf96d66b1133" exitCode=137 Jun 11 20:38:11.024923 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.024570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerDied","Data":"d17f5ada8eef616b1f8cc8b3f9536188a52cbb29e9931cc1e1a2e3fa4ab19d4e"} Jun 11 20:38:11.024923 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.024613 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerDied","Data":"f28b7e9dd507ea7bb0c2dbea9b4d2db31ba433148af7ed65521dbf96d66b1133"} Jun 11 20:38:11.504699 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.504672 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:38:11.541069 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.541039 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4666c749-9601-462c-a180-cd61392e53a7-feast-data\") pod \"4666c749-9601-462c-a180-cd61392e53a7\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " Jun 11 20:38:11.541257 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.541126 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-ui-tls\") pod \"4666c749-9601-462c-a180-cd61392e53a7\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " Jun 11 20:38:11.541257 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.541152 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-offline-tls\") pod \"4666c749-9601-462c-a180-cd61392e53a7\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " Jun 11 20:38:11.541257 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.541175 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-registry-tls\") pod \"4666c749-9601-462c-a180-cd61392e53a7\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " Jun 11 20:38:11.541425 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.541334 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5qbv\" (UniqueName: \"kubernetes.io/projected/4666c749-9601-462c-a180-cd61392e53a7-kube-api-access-r5qbv\") pod \"4666c749-9601-462c-a180-cd61392e53a7\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " Jun 11 20:38:11.541485 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.541424 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-online-tls\") pod \"4666c749-9601-462c-a180-cd61392e53a7\" (UID: \"4666c749-9601-462c-a180-cd61392e53a7\") " Jun 11 20:38:11.541731 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.541615 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4666c749-9601-462c-a180-cd61392e53a7-feast-data" (OuterVolumeSpecName: "feast-data") pod "4666c749-9601-462c-a180-cd61392e53a7" (UID: "4666c749-9601-462c-a180-cd61392e53a7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:38:11.541731 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.541687 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4666c749-9601-462c-a180-cd61392e53a7-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:38:11.543503 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.543474 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4666c749-9601-462c-a180-cd61392e53a7" (UID: "4666c749-9601-462c-a180-cd61392e53a7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:38:11.543749 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.543723 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4666c749-9601-462c-a180-cd61392e53a7" (UID: "4666c749-9601-462c-a180-cd61392e53a7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:38:11.543821 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.543768 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4666c749-9601-462c-a180-cd61392e53a7-kube-api-access-r5qbv" (OuterVolumeSpecName: "kube-api-access-r5qbv") pod "4666c749-9601-462c-a180-cd61392e53a7" (UID: "4666c749-9601-462c-a180-cd61392e53a7"). InnerVolumeSpecName "kube-api-access-r5qbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:38:11.543938 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.543918 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4666c749-9601-462c-a180-cd61392e53a7" (UID: "4666c749-9601-462c-a180-cd61392e53a7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:38:11.544069 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.543947 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-online-tls" (OuterVolumeSpecName: "online-tls") pod "4666c749-9601-462c-a180-cd61392e53a7" (UID: "4666c749-9601-462c-a180-cd61392e53a7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:38:11.642358 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.642269 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-ui-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:38:11.642358 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.642299 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:38:11.642358 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.642309 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:38:11.642358 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.642318 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qbv\" (UniqueName: \"kubernetes.io/projected/4666c749-9601-462c-a180-cd61392e53a7-kube-api-access-r5qbv\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:38:11.642358 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:11.642327 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4666c749-9601-462c-a180-cd61392e53a7-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:38:12.032070 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.032037 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" Jun 11 20:38:12.032519 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.032041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h" event={"ID":"4666c749-9601-462c-a180-cd61392e53a7","Type":"ContainerDied","Data":"2e6d6357230d1447859ed2b053e78aeebd02fd04489573af5eb77f037e152956"} Jun 11 20:38:12.032519 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.032172 2564 scope.go:117] "RemoveContainer" containerID="42484db78b65abdee5cc855512e1876923ffc4c764137f1532e6707e3237cf92" Jun 11 20:38:12.041694 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.041671 2564 scope.go:117] "RemoveContainer" containerID="d17f5ada8eef616b1f8cc8b3f9536188a52cbb29e9931cc1e1a2e3fa4ab19d4e" Jun 11 20:38:12.049810 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.049789 2564 scope.go:117] "RemoveContainer" containerID="d62db6f78ffcf93809798ef07cfd3422736705035fda68273448ae2698f351a2" Jun 11 20:38:12.054763 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.054738 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h"] Jun 11 20:38:12.057034 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.057009 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-7kn6h"] Jun 11 20:38:12.058827 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.058807 2564 scope.go:117] "RemoveContainer" containerID="f28b7e9dd507ea7bb0c2dbea9b4d2db31ba433148af7ed65521dbf96d66b1133" Jun 11 20:38:12.066923 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.066901 2564 scope.go:117] "RemoveContainer" containerID="07f8ed6f51310c8c314907bfa0efeb50cd152b2ff48c812ccb0830c3c3f3f89b" Jun 11 20:38:12.074581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:12.074562 2564 scope.go:117] "RemoveContainer" containerID="712b4edd2431192296a1a05d72be59bdb4945cf0e94654398e143caaa1289c7e" Jun 11 20:38:13.074925 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:38:13.074890 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4666c749-9601-462c-a180-cd61392e53a7" path="/var/lib/kubelet/pods/4666c749-9601-462c-a180-cd61392e53a7/volumes" Jun 11 20:39:02.785956 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.785863 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt"] Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786336 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="feast-apply" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786356 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="feast-apply" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786376 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="ui" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786384 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="ui" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786399 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="registry" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786407 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="registry" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786422 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="online" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786431 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="online" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786440 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="offline" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786447 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="offline" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786462 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="feast-init" Jun 11 20:39:02.786514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786470 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="feast-init" Jun 11 20:39:02.787135 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786557 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="registry" Jun 11 20:39:02.787135 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786570 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="ui" Jun 11 20:39:02.787135 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786581 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="online" Jun 11 20:39:02.787135 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.786590 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4666c749-9601-462c-a180-cd61392e53a7" containerName="offline" Jun 11 20:39:02.790314 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.790292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.792814 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.792791 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-vfdjb\"" Jun 11 20:39:02.793063 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.792804 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 11 20:39:02.793229 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.792805 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 11 20:39:02.793295 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.792842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 11 20:39:02.793359 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.792843 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 11 20:39:02.793410 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.792876 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 11 20:39:02.793465 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.792886 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 11 20:39:02.793530 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.792886 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jun 11 20:39:02.800261 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.800235 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt"] Jun 11 20:39:02.880587 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.880545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-offline-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.880587 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.880589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668fb\" (UniqueName: \"kubernetes.io/projected/5f2b340e-241f-4b9e-81ff-d6651d5818c0-kube-api-access-668fb\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.880791 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.880627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b340e-241f-4b9e-81ff-d6651d5818c0-feast-data\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.880791 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.880709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5f2b340e-241f-4b9e-81ff-d6651d5818c0-registry-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.880791 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.880759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-online-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.880791 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.880781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-ui-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.981992 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.981929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5f2b340e-241f-4b9e-81ff-d6651d5818c0-registry-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.982177 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.982015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-online-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.982177 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.982042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-ui-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.982285 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.982183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-offline-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.982285 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.982223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668fb\" (UniqueName: \"kubernetes.io/projected/5f2b340e-241f-4b9e-81ff-d6651d5818c0-kube-api-access-668fb\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.982285 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.982261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b340e-241f-4b9e-81ff-d6651d5818c0-feast-data\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.982660 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.982631 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b340e-241f-4b9e-81ff-d6651d5818c0-feast-data\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.982788 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.982771 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5f2b340e-241f-4b9e-81ff-d6651d5818c0-registry-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.984596 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.984561 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-online-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.984710 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.984614 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-offline-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.984710 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.984624 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-ui-tls\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:02.989487 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:02.989462 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668fb\" (UniqueName: \"kubernetes.io/projected/5f2b340e-241f-4b9e-81ff-d6651d5818c0-kube-api-access-668fb\") pod \"feast-simple-feast-remote-setup-5dcb57548c-twsqt\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:03.102389 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:03.102297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:03.240144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:03.240120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt"] Jun 11 20:39:03.242024 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:39:03.241997 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f2b340e_241f_4b9e_81ff_d6651d5818c0.slice/crio-2444b5a5b1e8341347811e488afca07e30f3c83c253808b857a98184b1fd552a WatchSource:0}: Error finding container 2444b5a5b1e8341347811e488afca07e30f3c83c253808b857a98184b1fd552a: Status 404 returned error can't find the container with id 2444b5a5b1e8341347811e488afca07e30f3c83c253808b857a98184b1fd552a Jun 11 20:39:04.246940 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:04.246906 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerStarted","Data":"2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe"} Jun 11 20:39:04.247522 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:04.246997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerStarted","Data":"2444b5a5b1e8341347811e488afca07e30f3c83c253808b857a98184b1fd552a"} Jun 11 20:39:08.269167 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:08.269129 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerID="2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe" exitCode=0 Jun 11 20:39:08.269584 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:08.269203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerDied","Data":"2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe"} Jun 11 20:39:09.275513 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:09.275481 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerStarted","Data":"7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49"} Jun 11 20:39:14.303565 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:14.303528 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerID="7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49" exitCode=0 Jun 11 20:39:14.303984 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:14.303577 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerDied","Data":"7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49"} Jun 11 20:39:15.312314 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:15.312277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerStarted","Data":"44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb"} Jun 11 20:39:15.312802 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:15.312327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerStarted","Data":"dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e"} Jun 11 20:39:15.312802 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:15.312342 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerStarted","Data":"bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02"} Jun 11 20:39:15.334508 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:15.334441 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podStartSLOduration=13.334419341 podStartE2EDuration="13.334419341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:39:15.332299712 +0000 UTC m=+820.687046048" watchObservedRunningTime="2026-06-11 20:39:15.334419341 +0000 UTC m=+820.689165658" Jun 11 20:39:18.102993 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:18.102937 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:18.103475 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:18.103020 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:18.103475 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:18.103038 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:18.104925 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:18.104884 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jun 11 20:39:18.104925 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:18.104907 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.28:8816: connect: connection refused" Jun 11 20:39:18.105146 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:18.104916 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jun 11 20:39:21.103848 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:21.103808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:21.104378 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:21.104245 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:21.104587 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:21.104562 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:21.104717 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:21.104618 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:21.104773 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:21.104756 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:21.104817 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:21.104794 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:21.108348 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:21.108324 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:21.347538 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:21.347412 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:39:21.350354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:39:21.350330 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:40:24.050612 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:24.050526 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt"] Jun 11 20:40:24.051124 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:24.050830 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="online" containerID="cri-o://bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02" gracePeriod=30 Jun 11 20:40:24.051124 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:24.051064 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="ui" containerID="cri-o://44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb" gracePeriod=30 Jun 11 20:40:24.051246 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:24.051178 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="offline" containerID="cri-o://dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e" gracePeriod=30 Jun 11 20:40:24.161133 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:24.161095 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq"] Jun 11 20:40:24.161418 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:24.161384 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="registry" containerID="cri-o://f305c357184aa05ddab0f549ea64abf7bf95f59f86673e96526775aa6fefe6c2" gracePeriod=30 Jun 11 20:40:24.161566 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:24.161490 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="offline" containerID="cri-o://efa8bd04b429e28e3c7dee468a63acc74ec4b8ad57a640fa27c03501f5e6708f" gracePeriod=30 Jun 11 20:40:24.161566 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:24.161498 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="ui" containerID="cri-o://21330bc8ec958f598701878b8d23eff73a0364997a48f5192d6d06d685a9836d" gracePeriod=30 Jun 11 20:40:24.161687 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:24.161561 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="online" containerID="cri-o://adc03fabfdc23fd1cf56902ecff5f08f136ae49203da71ded9d3b07f56195703" gracePeriod=30 Jun 11 20:40:25.615586 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:25.615549 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerID="44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb" exitCode=0 Jun 11 20:40:25.615586 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:25.615576 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerID="bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02" exitCode=0 Jun 11 20:40:25.616052 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:25.615622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerDied","Data":"44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb"} Jun 11 20:40:25.616052 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:25.615663 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerDied","Data":"bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02"} Jun 11 20:40:25.618529 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:25.618500 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerID="21330bc8ec958f598701878b8d23eff73a0364997a48f5192d6d06d685a9836d" exitCode=0 Jun 11 20:40:25.618529 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:25.618527 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerID="adc03fabfdc23fd1cf56902ecff5f08f136ae49203da71ded9d3b07f56195703" exitCode=0 Jun 11 20:40:25.618709 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:25.618568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerDied","Data":"21330bc8ec958f598701878b8d23eff73a0364997a48f5192d6d06d685a9836d"} Jun 11 20:40:25.618709 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:25.618603 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerDied","Data":"adc03fabfdc23fd1cf56902ecff5f08f136ae49203da71ded9d3b07f56195703"} Jun 11 20:40:30.976474 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:30.976428 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Jun 11 20:40:30.976474 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:30.976452 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Jun 11 20:40:31.104458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:31.104407 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jun 11 20:40:31.348651 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:31.348604 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jun 11 20:40:35.083390 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:35.083361 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:40:35.084949 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:35.084928 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:40:35.089787 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:35.089766 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:40:35.091784 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:35.091762 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:40:40.976685 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:40.976627 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Jun 11 20:40:40.977215 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:40.976629 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Jun 11 20:40:41.104497 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:41.104448 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jun 11 20:40:41.348457 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:41.348412 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jun 11 20:40:50.976308 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:50.976259 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" Jun 11 20:40:50.978676 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:50.976263 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Jun 11 20:40:50.978676 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:50.976398 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:40:50.978676 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:50.976452 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:40:51.104900 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:51.104861 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jun 11 20:40:51.105102 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:51.105009 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:40:51.348171 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:51.348134 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jun 11 20:40:51.348345 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:51.348247 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:40:54.279619 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.279590 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:40:54.344493 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.344391 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-offline-tls\") pod \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " Jun 11 20:40:54.344493 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.344462 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5f2b340e-241f-4b9e-81ff-d6651d5818c0-registry-tls\") pod \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " Jun 11 20:40:54.344721 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.344495 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-online-tls\") pod \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " Jun 11 20:40:54.344721 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.344526 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-ui-tls\") pod \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " Jun 11 20:40:54.344721 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.344585 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b340e-241f-4b9e-81ff-d6651d5818c0-feast-data\") pod \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " Jun 11 20:40:54.344721 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.344613 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-668fb\" (UniqueName: \"kubernetes.io/projected/5f2b340e-241f-4b9e-81ff-d6651d5818c0-kube-api-access-668fb\") pod \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\" (UID: \"5f2b340e-241f-4b9e-81ff-d6651d5818c0\") " Jun 11 20:40:54.344924 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.344898 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2b340e-241f-4b9e-81ff-d6651d5818c0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5f2b340e-241f-4b9e-81ff-d6651d5818c0" (UID: "5f2b340e-241f-4b9e-81ff-d6651d5818c0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:40:54.345150 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.345119 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2b340e-241f-4b9e-81ff-d6651d5818c0-feast-data" (OuterVolumeSpecName: "feast-data") pod "5f2b340e-241f-4b9e-81ff-d6651d5818c0" (UID: "5f2b340e-241f-4b9e-81ff-d6651d5818c0"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:40:54.346852 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.346818 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5f2b340e-241f-4b9e-81ff-d6651d5818c0" (UID: "5f2b340e-241f-4b9e-81ff-d6651d5818c0"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:40:54.346852 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.346831 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5f2b340e-241f-4b9e-81ff-d6651d5818c0" (UID: "5f2b340e-241f-4b9e-81ff-d6651d5818c0"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:40:54.347020 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.346850 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2b340e-241f-4b9e-81ff-d6651d5818c0-kube-api-access-668fb" (OuterVolumeSpecName: "kube-api-access-668fb") pod "5f2b340e-241f-4b9e-81ff-d6651d5818c0" (UID: "5f2b340e-241f-4b9e-81ff-d6651d5818c0"). InnerVolumeSpecName "kube-api-access-668fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:40:54.347020 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.346881 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-online-tls" (OuterVolumeSpecName: "online-tls") pod "5f2b340e-241f-4b9e-81ff-d6651d5818c0" (UID: "5f2b340e-241f-4b9e-81ff-d6651d5818c0"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:40:54.445741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.445688 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-ui-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.445741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.445732 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2b340e-241f-4b9e-81ff-d6651d5818c0-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.445741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.445743 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668fb\" (UniqueName: \"kubernetes.io/projected/5f2b340e-241f-4b9e-81ff-d6651d5818c0-kube-api-access-668fb\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.445741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.445752 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.445741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.445760 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5f2b340e-241f-4b9e-81ff-d6651d5818c0-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.446150 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.445769 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f2b340e-241f-4b9e-81ff-d6651d5818c0-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.748780 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.748737 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerID="dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e" exitCode=137 Jun 11 20:40:54.748943 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.748811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerDied","Data":"dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e"} Jun 11 20:40:54.748943 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.748868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" event={"ID":"5f2b340e-241f-4b9e-81ff-d6651d5818c0","Type":"ContainerDied","Data":"2444b5a5b1e8341347811e488afca07e30f3c83c253808b857a98184b1fd552a"} Jun 11 20:40:54.748943 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.748815 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt" Jun 11 20:40:54.748943 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.748898 2564 scope.go:117] "RemoveContainer" containerID="44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb" Jun 11 20:40:54.753086 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.753058 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerID="efa8bd04b429e28e3c7dee468a63acc74ec4b8ad57a640fa27c03501f5e6708f" exitCode=137 Jun 11 20:40:54.753086 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.753083 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerID="f305c357184aa05ddab0f549ea64abf7bf95f59f86673e96526775aa6fefe6c2" exitCode=137 Jun 11 20:40:54.753402 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.753143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerDied","Data":"efa8bd04b429e28e3c7dee468a63acc74ec4b8ad57a640fa27c03501f5e6708f"} Jun 11 20:40:54.753402 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.753185 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerDied","Data":"f305c357184aa05ddab0f549ea64abf7bf95f59f86673e96526775aa6fefe6c2"} Jun 11 20:40:54.760900 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.760817 2564 scope.go:117] "RemoveContainer" containerID="dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e" Jun 11 20:40:54.771250 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.771232 2564 scope.go:117] "RemoveContainer" containerID="bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02" Jun 11 20:40:54.775139 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.775111 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt"] Jun 11 20:40:54.780396 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.780369 2564 scope.go:117] "RemoveContainer" containerID="7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49" Jun 11 20:40:54.780579 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.780559 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5dcb57548c-twsqt"] Jun 11 20:40:54.788367 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.788344 2564 scope.go:117] "RemoveContainer" containerID="2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe" Jun 11 20:40:54.797636 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.797617 2564 scope.go:117] "RemoveContainer" containerID="44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb" Jun 11 20:40:54.797980 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:40:54.797941 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb\": container with ID starting with 44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb not found: ID does not exist" containerID="44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb" Jun 11 20:40:54.798048 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.797990 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb"} err="failed to get container status \"44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb\": rpc error: code = NotFound desc = could not find container \"44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb\": container with ID starting with 44a281bf17148d7e0e0585bf592ee97e60b6cb6733a4bd96b0c41d464c2dfecb not found: ID does not exist" Jun 11 20:40:54.798048 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.798013 2564 scope.go:117] "RemoveContainer" containerID="dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e" Jun 11 20:40:54.798292 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:40:54.798274 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e\": container with ID starting with dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e not found: ID does not exist" containerID="dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e" Jun 11 20:40:54.798363 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.798313 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e"} err="failed to get container status \"dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e\": rpc error: code = NotFound desc = could not find container \"dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e\": container with ID starting with dcf5f660896d4afdeecbbf56a009fa49d20bba7f6ee68308a2833c93abd80d7e not found: ID does not exist" Jun 11 20:40:54.798363 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.798337 2564 scope.go:117] "RemoveContainer" containerID="bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02" Jun 11 20:40:54.798631 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:40:54.798614 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02\": container with ID starting with bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02 not found: ID does not exist" containerID="bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02" Jun 11 20:40:54.798675 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.798636 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02"} err="failed to get container status \"bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02\": rpc error: code = NotFound desc = could not find container \"bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02\": container with ID starting with bef9910afd40c9f469f70ffac28f99521dc9536a96ac7c5c91cd46db5ee24b02 not found: ID does not exist" Jun 11 20:40:54.798675 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.798651 2564 scope.go:117] "RemoveContainer" containerID="7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49" Jun 11 20:40:54.798891 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:40:54.798873 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49\": container with ID starting with 7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49 not found: ID does not exist" containerID="7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49" Jun 11 20:40:54.798987 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.798909 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49"} err="failed to get container status \"7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49\": rpc error: code = NotFound desc = could not find container \"7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49\": container with ID starting with 7db01b27b2ef031e245a47acfae4ae037594ecc406d2fdd6682c2ef7fc28cb49 not found: ID does not exist" Jun 11 20:40:54.798987 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.798929 2564 scope.go:117] "RemoveContainer" containerID="2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe" Jun 11 20:40:54.799220 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:40:54.799203 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe\": container with ID starting with 2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe not found: ID does not exist" containerID="2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe" Jun 11 20:40:54.799259 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.799233 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe"} err="failed to get container status \"2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe\": rpc error: code = NotFound desc = could not find container \"2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe\": container with ID starting with 2b70311955e4abdce16ce674fcd90e199576fc38adfd3e8f94983f79af0d0cfe not found: ID does not exist" Jun 11 20:40:54.816919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.816895 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:40:54.848894 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.848867 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-registry-tls\") pod \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " Jun 11 20:40:54.849079 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.848900 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a49ccc-a500-4cba-a668-e87473dcfcd9-feast-data\") pod \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " Jun 11 20:40:54.849079 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.848937 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q22mz\" (UniqueName: \"kubernetes.io/projected/f0a49ccc-a500-4cba-a668-e87473dcfcd9-kube-api-access-q22mz\") pod \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " Jun 11 20:40:54.849079 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.848995 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-offline-tls\") pod \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " Jun 11 20:40:54.849079 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.849029 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-online-tls\") pod \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " Jun 11 20:40:54.849079 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.849069 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-ui-tls\") pod \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\" (UID: \"f0a49ccc-a500-4cba-a668-e87473dcfcd9\") " Jun 11 20:40:54.849537 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.849501 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a49ccc-a500-4cba-a668-e87473dcfcd9-feast-data" (OuterVolumeSpecName: "feast-data") pod "f0a49ccc-a500-4cba-a668-e87473dcfcd9" (UID: "f0a49ccc-a500-4cba-a668-e87473dcfcd9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:40:54.851309 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.851284 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f0a49ccc-a500-4cba-a668-e87473dcfcd9" (UID: "f0a49ccc-a500-4cba-a668-e87473dcfcd9"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:40:54.851557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.851536 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-online-tls" (OuterVolumeSpecName: "online-tls") pod "f0a49ccc-a500-4cba-a668-e87473dcfcd9" (UID: "f0a49ccc-a500-4cba-a668-e87473dcfcd9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:40:54.851637 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.851601 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f0a49ccc-a500-4cba-a668-e87473dcfcd9" (UID: "f0a49ccc-a500-4cba-a668-e87473dcfcd9"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:40:54.851689 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.851636 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f0a49ccc-a500-4cba-a668-e87473dcfcd9" (UID: "f0a49ccc-a500-4cba-a668-e87473dcfcd9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:40:54.851689 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.851656 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a49ccc-a500-4cba-a668-e87473dcfcd9-kube-api-access-q22mz" (OuterVolumeSpecName: "kube-api-access-q22mz") pod "f0a49ccc-a500-4cba-a668-e87473dcfcd9" (UID: "f0a49ccc-a500-4cba-a668-e87473dcfcd9"). InnerVolumeSpecName "kube-api-access-q22mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:40:54.950210 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.950117 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.950210 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.950154 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.950210 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.950163 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-ui-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.950210 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.950171 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f0a49ccc-a500-4cba-a668-e87473dcfcd9-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.950210 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.950178 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a49ccc-a500-4cba-a668-e87473dcfcd9-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:54.950210 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:54.950186 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q22mz\" (UniqueName: \"kubernetes.io/projected/f0a49ccc-a500-4cba-a668-e87473dcfcd9-kube-api-access-q22mz\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:40:55.076456 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.076416 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" path="/var/lib/kubelet/pods/5f2b340e-241f-4b9e-81ff-d6651d5818c0/volumes" Jun 11 20:40:55.760560 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.760528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" event={"ID":"f0a49ccc-a500-4cba-a668-e87473dcfcd9","Type":"ContainerDied","Data":"0d097b36e15bfb7b88c23dead856180e7917343982f8c486e5a2d984cce7a977"} Jun 11 20:40:55.761023 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.760575 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq" Jun 11 20:40:55.761023 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.760584 2564 scope.go:117] "RemoveContainer" containerID="21330bc8ec958f598701878b8d23eff73a0364997a48f5192d6d06d685a9836d" Jun 11 20:40:55.769517 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.769498 2564 scope.go:117] "RemoveContainer" containerID="efa8bd04b429e28e3c7dee468a63acc74ec4b8ad57a640fa27c03501f5e6708f" Jun 11 20:40:55.777680 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.777651 2564 scope.go:117] "RemoveContainer" containerID="adc03fabfdc23fd1cf56902ecff5f08f136ae49203da71ded9d3b07f56195703" Jun 11 20:40:55.779494 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.779467 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq"] Jun 11 20:40:55.784684 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.784660 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-846f56fdd8-cg2wq"] Jun 11 20:40:55.786912 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.786888 2564 scope.go:117] "RemoveContainer" containerID="f305c357184aa05ddab0f549ea64abf7bf95f59f86673e96526775aa6fefe6c2" Jun 11 20:40:55.794416 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.794396 2564 scope.go:117] "RemoveContainer" containerID="f663cfa0141ee0b99aa66a3d5550ab9782bd6df6fe4b875b5b474525c775c6c5" Jun 11 20:40:55.801607 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:55.801588 2564 scope.go:117] "RemoveContainer" containerID="f96a0cc9157ce736a9858bef613f095ef1f92aa4f8608c9759c6d3c14430f816" Jun 11 20:40:57.075001 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:40:57.074945 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" path="/var/lib/kubelet/pods/f0a49ccc-a500-4cba-a668-e87473dcfcd9/volumes" Jun 11 20:41:06.880686 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.880649 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-6mv8z"] Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.880950 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="ui" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.880976 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="ui" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.880993 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="online" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.880999 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="online" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881009 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="offline" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881015 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="offline" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881021 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="ui" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881026 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="ui" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881036 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="feast-apply" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881041 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="feast-apply" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881046 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="feast-init" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881051 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="feast-init" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881057 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="online" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881062 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="online" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881067 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="feast-init" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881073 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="feast-init" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881080 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="registry" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881085 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="registry" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881091 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="offline" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881096 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="offline" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881104 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="feast-apply" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881109 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="feast-apply" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881158 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="online" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881167 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="offline" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881173 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="ui" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881178 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="online" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881185 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="ui" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881191 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a49ccc-a500-4cba-a668-e87473dcfcd9" containerName="registry" Jun 11 20:41:06.881200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.881199 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2b340e-241f-4b9e-81ff-d6651d5818c0" containerName="offline" Jun 11 20:41:06.884273 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.884252 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:41:06.886411 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.886375 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jun 11 20:41:06.886545 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.886377 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 11 20:41:06.886545 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.886377 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-zc2bd\"" Jun 11 20:41:06.886923 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.886897 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 11 20:41:06.889528 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.889501 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6mv8z"] Jun 11 20:41:06.946328 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.946291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc79m\" (UniqueName: \"kubernetes.io/projected/edc83486-f597-4c67-94dc-82e7b0906a45-kube-api-access-xc79m\") pod \"postgres-864846d4f5-6mv8z\" (UID: \"edc83486-f597-4c67-94dc-82e7b0906a45\") " pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:41:06.946515 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:06.946363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edc83486-f597-4c67-94dc-82e7b0906a45-postgresdata\") pod \"postgres-864846d4f5-6mv8z\" (UID: \"edc83486-f597-4c67-94dc-82e7b0906a45\") " pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:41:07.047507 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:07.047448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc79m\" (UniqueName: \"kubernetes.io/projected/edc83486-f597-4c67-94dc-82e7b0906a45-kube-api-access-xc79m\") pod \"postgres-864846d4f5-6mv8z\" (UID: \"edc83486-f597-4c67-94dc-82e7b0906a45\") " pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:41:07.047696 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:07.047546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edc83486-f597-4c67-94dc-82e7b0906a45-postgresdata\") pod \"postgres-864846d4f5-6mv8z\" (UID: \"edc83486-f597-4c67-94dc-82e7b0906a45\") " pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:41:07.047924 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:07.047906 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edc83486-f597-4c67-94dc-82e7b0906a45-postgresdata\") pod \"postgres-864846d4f5-6mv8z\" (UID: \"edc83486-f597-4c67-94dc-82e7b0906a45\") " pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:41:07.056274 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:07.056239 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc79m\" (UniqueName: \"kubernetes.io/projected/edc83486-f597-4c67-94dc-82e7b0906a45-kube-api-access-xc79m\") pod \"postgres-864846d4f5-6mv8z\" (UID: \"edc83486-f597-4c67-94dc-82e7b0906a45\") " pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:41:07.207321 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:07.207226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:41:07.342690 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:07.342510 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6mv8z"] Jun 11 20:41:07.345517 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:41:07.345487 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedc83486_f597_4c67_94dc_82e7b0906a45.slice/crio-3f0ee4a121af798089da7b8d833174551ed7ceced920c7d57708be5a9321e6bb WatchSource:0}: Error finding container 3f0ee4a121af798089da7b8d833174551ed7ceced920c7d57708be5a9321e6bb: Status 404 returned error can't find the container with id 3f0ee4a121af798089da7b8d833174551ed7ceced920c7d57708be5a9321e6bb Jun 11 20:41:07.347807 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:07.347784 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 11 20:41:07.812583 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:07.812547 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-6mv8z" event={"ID":"edc83486-f597-4c67-94dc-82e7b0906a45","Type":"ContainerStarted","Data":"3f0ee4a121af798089da7b8d833174551ed7ceced920c7d57708be5a9321e6bb"} Jun 11 20:41:14.844912 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:14.844872 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-6mv8z" event={"ID":"edc83486-f597-4c67-94dc-82e7b0906a45","Type":"ContainerStarted","Data":"29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81"} Jun 11 20:41:14.860492 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:14.860434 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-6mv8z" podStartSLOduration=2.406397046 podStartE2EDuration="8.860417624s" totalImagesPullingTime="6.454020578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:41:14.858445968 +0000 UTC m=+940.213192304" watchObservedRunningTime="2026-06-11 20:41:14.860417624 +0000 UTC m=+940.215163940" Jun 11 20:41:17.558840 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.558798 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-f67c75994-nmfjc"] Jun 11 20:41:17.564887 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.564861 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.567343 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.567311 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jun 11 20:41:17.567343 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.567331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-nz2vf\"" Jun 11 20:41:17.567577 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.567353 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jun 11 20:41:17.567577 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.567503 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jun 11 20:41:17.567757 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.567611 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jun 11 20:41:17.573608 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.573580 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-f67c75994-nmfjc"] Jun 11 20:41:17.640520 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.640479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-offline-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.640708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.640551 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt25n\" (UniqueName: \"kubernetes.io/projected/5e33b670-d893-4a18-933b-35e69f6a581f-kube-api-access-tt25n\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.640708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.640595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-registry-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.640708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.640687 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-online-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.640832 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.640714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e33b670-d893-4a18-933b-35e69f6a581f-feast-data\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.741241 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.741197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt25n\" (UniqueName: \"kubernetes.io/projected/5e33b670-d893-4a18-933b-35e69f6a581f-kube-api-access-tt25n\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.741436 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.741262 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-registry-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.741436 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.741310 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-online-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.741436 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.741333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e33b670-d893-4a18-933b-35e69f6a581f-feast-data\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.741436 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.741364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-offline-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.741436 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:41:17.741421 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Jun 11 20:41:17.741732 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:41:17.741497 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-online-tls podName:5e33b670-d893-4a18-933b-35e69f6a581f nodeName:}" failed. No retries permitted until 2026-06-11 20:41:18.2414744 +0000 UTC m=+943.596220711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-online-tls") pod "feast-credit-scoring-f67c75994-nmfjc" (UID: "5e33b670-d893-4a18-933b-35e69f6a581f") : secret "feast-credit-scoring-online-tls" not found Jun 11 20:41:17.741803 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.741778 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e33b670-d893-4a18-933b-35e69f6a581f-feast-data\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.743762 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.743743 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-registry-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.743824 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.743763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-offline-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:17.749887 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:17.749862 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt25n\" (UniqueName: \"kubernetes.io/projected/5e33b670-d893-4a18-933b-35e69f6a581f-kube-api-access-tt25n\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:18.245576 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:18.245536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-online-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:18.248015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:18.247953 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-online-tls\") pod \"feast-credit-scoring-f67c75994-nmfjc\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:18.477117 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:18.477071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:18.617011 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:18.616986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-f67c75994-nmfjc"] Jun 11 20:41:18.618547 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:41:18.618516 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e33b670_d893_4a18_933b_35e69f6a581f.slice/crio-c74af3129a3492b8d879d81b16e90bc748425646988c8b91ab5e63add96ff258 WatchSource:0}: Error finding container c74af3129a3492b8d879d81b16e90bc748425646988c8b91ab5e63add96ff258: Status 404 returned error can't find the container with id c74af3129a3492b8d879d81b16e90bc748425646988c8b91ab5e63add96ff258 Jun 11 20:41:18.866675 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:18.866573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerStarted","Data":"149af3f8eded2eec0795a3b78e1573b33bd55e07f2ac9e79e2ef1a2cfec8d0b1"} Jun 11 20:41:18.866675 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:18.866655 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerStarted","Data":"c74af3129a3492b8d879d81b16e90bc748425646988c8b91ab5e63add96ff258"} Jun 11 20:41:19.872360 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:19.872278 2564 generic.go:356] "Generic (PLEG): container finished" podID="5e33b670-d893-4a18-933b-35e69f6a581f" containerID="149af3f8eded2eec0795a3b78e1573b33bd55e07f2ac9e79e2ef1a2cfec8d0b1" exitCode=0 Jun 11 20:41:19.872705 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:19.872371 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerDied","Data":"149af3f8eded2eec0795a3b78e1573b33bd55e07f2ac9e79e2ef1a2cfec8d0b1"} Jun 11 20:41:20.879039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:20.879001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerStarted","Data":"16b05d74fe8be9409065cf51bbf77d33ceb7de9871a31e6b5322d894f468f31c"} Jun 11 20:41:26.910758 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:26.910652 2564 generic.go:356] "Generic (PLEG): container finished" podID="5e33b670-d893-4a18-933b-35e69f6a581f" containerID="16b05d74fe8be9409065cf51bbf77d33ceb7de9871a31e6b5322d894f468f31c" exitCode=0 Jun 11 20:41:26.910758 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:26.910708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerDied","Data":"16b05d74fe8be9409065cf51bbf77d33ceb7de9871a31e6b5322d894f468f31c"} Jun 11 20:41:27.929592 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:27.929549 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerStarted","Data":"35b34d9c831f727afcf7785b48f367ae51b1ba5e67f74c7f47b87c25b1083f13"} Jun 11 20:41:27.930000 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:27.929667 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerStarted","Data":"f838155f4e77816e6e66cfcd40d00e72e9a0e0c2aa9576b18a4afe052cfe10a5"} Jun 11 20:41:27.930000 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:27.929697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerStarted","Data":"d9ae394c6959f4186c6b68b714f1cb5221ce8510b7eb2196acd5fd8c2e4115b7"} Jun 11 20:41:27.953959 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:27.953902 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podStartSLOduration=10.953879484 podStartE2EDuration="10.953879484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:41:27.951348795 +0000 UTC m=+953.306095111" watchObservedRunningTime="2026-06-11 20:41:27.953879484 +0000 UTC m=+953.308625801" Jun 11 20:41:28.477370 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:28.477323 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:28.477542 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:28.477415 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:28.477542 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:28.477443 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:30.478274 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:30.478228 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:30.478760 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:30.478308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:30.478760 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:30.478333 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:30.479994 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:30.479936 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Jun 11 20:41:30.479994 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:30.479953 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" Jun 11 20:41:30.480168 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:30.480019 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jun 11 20:41:33.478331 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:33.478282 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Jun 11 20:41:33.479202 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:33.479178 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:33.480049 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:33.480029 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:33.486062 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:33.486036 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:33.965827 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:33.965797 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:36.478039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:36.477989 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Jun 11 20:41:39.478758 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:39.478725 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:41:39.479449 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:41:39.479429 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:42:43.954046 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:43.954004 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-f67c75994-nmfjc"] Jun 11 20:42:43.954593 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:43.954312 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="registry" containerID="cri-o://d9ae394c6959f4186c6b68b714f1cb5221ce8510b7eb2196acd5fd8c2e4115b7" gracePeriod=30 Jun 11 20:42:43.954593 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:43.954344 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="online" containerID="cri-o://f838155f4e77816e6e66cfcd40d00e72e9a0e0c2aa9576b18a4afe052cfe10a5" gracePeriod=30 Jun 11 20:42:43.954593 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:43.954391 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="offline" containerID="cri-o://35b34d9c831f727afcf7785b48f367ae51b1ba5e67f74c7f47b87c25b1083f13" gracePeriod=30 Jun 11 20:42:43.968341 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:43.968299 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6mv8z"] Jun 11 20:42:43.968530 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:43.968494 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-6mv8z" podUID="edc83486-f597-4c67-94dc-82e7b0906a45" containerName="postgres" containerID="cri-o://29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81" gracePeriod=30 Jun 11 20:42:44.218904 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.218876 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:42:44.260432 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.260394 2564 generic.go:356] "Generic (PLEG): container finished" podID="edc83486-f597-4c67-94dc-82e7b0906a45" containerID="29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81" exitCode=0 Jun 11 20:42:44.260616 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.260434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-6mv8z" event={"ID":"edc83486-f597-4c67-94dc-82e7b0906a45","Type":"ContainerDied","Data":"29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81"} Jun 11 20:42:44.260616 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.260475 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-6mv8z" Jun 11 20:42:44.260616 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.260499 2564 scope.go:117] "RemoveContainer" containerID="29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81" Jun 11 20:42:44.260616 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.260485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-6mv8z" event={"ID":"edc83486-f597-4c67-94dc-82e7b0906a45","Type":"ContainerDied","Data":"3f0ee4a121af798089da7b8d833174551ed7ceced920c7d57708be5a9321e6bb"} Jun 11 20:42:44.270885 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.270854 2564 scope.go:117] "RemoveContainer" containerID="29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81" Jun 11 20:42:44.271230 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:42:44.271198 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81\": container with ID starting with 29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81 not found: ID does not exist" containerID="29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81" Jun 11 20:42:44.271323 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.271230 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81"} err="failed to get container status \"29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81\": rpc error: code = NotFound desc = could not find container \"29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81\": container with ID starting with 29588d031280289a9514928e0c33ae3e5081e3f4a0565dff08f5aa59092a5d81 not found: ID does not exist" Jun 11 20:42:44.300564 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.300531 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc79m\" (UniqueName: \"kubernetes.io/projected/edc83486-f597-4c67-94dc-82e7b0906a45-kube-api-access-xc79m\") pod \"edc83486-f597-4c67-94dc-82e7b0906a45\" (UID: \"edc83486-f597-4c67-94dc-82e7b0906a45\") " Jun 11 20:42:44.300820 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.300603 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edc83486-f597-4c67-94dc-82e7b0906a45-postgresdata\") pod \"edc83486-f597-4c67-94dc-82e7b0906a45\" (UID: \"edc83486-f597-4c67-94dc-82e7b0906a45\") " Jun 11 20:42:44.302775 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.302741 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc83486-f597-4c67-94dc-82e7b0906a45-kube-api-access-xc79m" (OuterVolumeSpecName: "kube-api-access-xc79m") pod "edc83486-f597-4c67-94dc-82e7b0906a45" (UID: "edc83486-f597-4c67-94dc-82e7b0906a45"). InnerVolumeSpecName "kube-api-access-xc79m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:42:44.334167 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.334113 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc83486-f597-4c67-94dc-82e7b0906a45-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "edc83486-f597-4c67-94dc-82e7b0906a45" (UID: "edc83486-f597-4c67-94dc-82e7b0906a45"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:42:44.402174 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.402122 2564 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/edc83486-f597-4c67-94dc-82e7b0906a45-postgresdata\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:42:44.402174 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.402161 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc79m\" (UniqueName: \"kubernetes.io/projected/edc83486-f597-4c67-94dc-82e7b0906a45-kube-api-access-xc79m\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:42:44.583093 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.583055 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6mv8z"] Jun 11 20:42:44.586973 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:44.586928 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6mv8z"] Jun 11 20:42:45.075453 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:45.075412 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc83486-f597-4c67-94dc-82e7b0906a45" path="/var/lib/kubelet/pods/edc83486-f597-4c67-94dc-82e7b0906a45/volumes" Jun 11 20:42:45.268600 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:45.268566 2564 generic.go:356] "Generic (PLEG): container finished" podID="5e33b670-d893-4a18-933b-35e69f6a581f" containerID="f838155f4e77816e6e66cfcd40d00e72e9a0e0c2aa9576b18a4afe052cfe10a5" exitCode=0 Jun 11 20:42:45.268776 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:45.268637 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerDied","Data":"f838155f4e77816e6e66cfcd40d00e72e9a0e0c2aa9576b18a4afe052cfe10a5"} Jun 11 20:42:53.963524 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:42:53.963480 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jun 11 20:43:03.963470 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:03.963422 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jun 11 20:43:13.962866 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:13.962817 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jun 11 20:43:13.963285 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:13.963033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:43:14.391642 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.391607 2564 generic.go:356] "Generic (PLEG): container finished" podID="5e33b670-d893-4a18-933b-35e69f6a581f" containerID="35b34d9c831f727afcf7785b48f367ae51b1ba5e67f74c7f47b87c25b1083f13" exitCode=137 Jun 11 20:43:14.391642 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.391633 2564 generic.go:356] "Generic (PLEG): container finished" podID="5e33b670-d893-4a18-933b-35e69f6a581f" containerID="d9ae394c6959f4186c6b68b714f1cb5221ce8510b7eb2196acd5fd8c2e4115b7" exitCode=137 Jun 11 20:43:14.391853 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.391678 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerDied","Data":"35b34d9c831f727afcf7785b48f367ae51b1ba5e67f74c7f47b87c25b1083f13"} Jun 11 20:43:14.391853 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.391730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerDied","Data":"d9ae394c6959f4186c6b68b714f1cb5221ce8510b7eb2196acd5fd8c2e4115b7"} Jun 11 20:43:14.607607 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.607582 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:43:14.765610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.765575 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e33b670-d893-4a18-933b-35e69f6a581f-feast-data\") pod \"5e33b670-d893-4a18-933b-35e69f6a581f\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " Jun 11 20:43:14.765806 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.765624 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt25n\" (UniqueName: \"kubernetes.io/projected/5e33b670-d893-4a18-933b-35e69f6a581f-kube-api-access-tt25n\") pod \"5e33b670-d893-4a18-933b-35e69f6a581f\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " Jun 11 20:43:14.765806 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.765674 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-online-tls\") pod \"5e33b670-d893-4a18-933b-35e69f6a581f\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " Jun 11 20:43:14.765806 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.765728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-offline-tls\") pod \"5e33b670-d893-4a18-933b-35e69f6a581f\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " Jun 11 20:43:14.765806 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.765749 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-registry-tls\") pod \"5e33b670-d893-4a18-933b-35e69f6a581f\" (UID: \"5e33b670-d893-4a18-933b-35e69f6a581f\") " Jun 11 20:43:14.767945 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.767910 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e33b670-d893-4a18-933b-35e69f6a581f-kube-api-access-tt25n" (OuterVolumeSpecName: "kube-api-access-tt25n") pod "5e33b670-d893-4a18-933b-35e69f6a581f" (UID: "5e33b670-d893-4a18-933b-35e69f6a581f"). InnerVolumeSpecName "kube-api-access-tt25n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:43:14.768115 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.767945 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5e33b670-d893-4a18-933b-35e69f6a581f" (UID: "5e33b670-d893-4a18-933b-35e69f6a581f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:43:14.768201 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.768185 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5e33b670-d893-4a18-933b-35e69f6a581f" (UID: "5e33b670-d893-4a18-933b-35e69f6a581f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:43:14.768318 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.768302 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-online-tls" (OuterVolumeSpecName: "online-tls") pod "5e33b670-d893-4a18-933b-35e69f6a581f" (UID: "5e33b670-d893-4a18-933b-35e69f6a581f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:43:14.773141 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.773112 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e33b670-d893-4a18-933b-35e69f6a581f-feast-data" (OuterVolumeSpecName: "feast-data") pod "5e33b670-d893-4a18-933b-35e69f6a581f" (UID: "5e33b670-d893-4a18-933b-35e69f6a581f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:43:14.866436 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.866392 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:43:14.866436 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.866425 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:43:14.866436 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.866434 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e33b670-d893-4a18-933b-35e69f6a581f-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:43:14.866436 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.866443 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt25n\" (UniqueName: \"kubernetes.io/projected/5e33b670-d893-4a18-933b-35e69f6a581f-kube-api-access-tt25n\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:43:14.866697 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:14.866453 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e33b670-d893-4a18-933b-35e69f6a581f-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:43:15.399070 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:15.398987 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" Jun 11 20:43:15.399070 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:15.398989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-f67c75994-nmfjc" event={"ID":"5e33b670-d893-4a18-933b-35e69f6a581f","Type":"ContainerDied","Data":"c74af3129a3492b8d879d81b16e90bc748425646988c8b91ab5e63add96ff258"} Jun 11 20:43:15.399070 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:15.399057 2564 scope.go:117] "RemoveContainer" containerID="35b34d9c831f727afcf7785b48f367ae51b1ba5e67f74c7f47b87c25b1083f13" Jun 11 20:43:15.408407 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:15.408385 2564 scope.go:117] "RemoveContainer" containerID="f838155f4e77816e6e66cfcd40d00e72e9a0e0c2aa9576b18a4afe052cfe10a5" Jun 11 20:43:15.416254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:15.416225 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-f67c75994-nmfjc"] Jun 11 20:43:15.417245 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:15.417219 2564 scope.go:117] "RemoveContainer" containerID="d9ae394c6959f4186c6b68b714f1cb5221ce8510b7eb2196acd5fd8c2e4115b7" Jun 11 20:43:15.419441 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:15.419418 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-f67c75994-nmfjc"] Jun 11 20:43:15.425974 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:15.425942 2564 scope.go:117] "RemoveContainer" containerID="16b05d74fe8be9409065cf51bbf77d33ceb7de9871a31e6b5322d894f468f31c" Jun 11 20:43:15.434075 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:15.434027 2564 scope.go:117] "RemoveContainer" containerID="149af3f8eded2eec0795a3b78e1573b33bd55e07f2ac9e79e2ef1a2cfec8d0b1" Jun 11 20:43:17.079876 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:43:17.079837 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" path="/var/lib/kubelet/pods/5e33b670-d893-4a18-933b-35e69f6a581f/volumes" Jun 11 20:45:00.350388 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350302 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-w7vsd"] Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350607 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="registry" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350619 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="registry" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350628 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="online" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350633 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="online" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350642 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="feast-init" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350647 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="feast-init" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350656 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="feast-apply" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350661 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="feast-apply" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350672 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc83486-f597-4c67-94dc-82e7b0906a45" containerName="postgres" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350677 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc83486-f597-4c67-94dc-82e7b0906a45" containerName="postgres" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350684 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="offline" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350689 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="offline" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350746 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="offline" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350755 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="registry" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350762 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e33b670-d893-4a18-933b-35e69f6a581f" containerName="online" Jun 11 20:45:00.350834 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.350768 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc83486-f597-4c67-94dc-82e7b0906a45" containerName="postgres" Jun 11 20:45:00.353467 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.353450 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" Jun 11 20:45:00.355719 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.355683 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jun 11 20:45:00.356428 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.356399 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-jt6vv\"" Jun 11 20:45:00.356428 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.356418 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jun 11 20:45:00.369275 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.369249 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-w7vsd"] Jun 11 20:45:00.431300 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.431263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wcr4\" (UniqueName: \"kubernetes.io/projected/499ac8e2-b1d1-441b-8c03-49acfcd8f79a-kube-api-access-2wcr4\") pod \"redis-69bbd798b6-w7vsd\" (UID: \"499ac8e2-b1d1-441b-8c03-49acfcd8f79a\") " pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" Jun 11 20:45:00.532319 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.532277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wcr4\" (UniqueName: \"kubernetes.io/projected/499ac8e2-b1d1-441b-8c03-49acfcd8f79a-kube-api-access-2wcr4\") pod \"redis-69bbd798b6-w7vsd\" (UID: \"499ac8e2-b1d1-441b-8c03-49acfcd8f79a\") " pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" Jun 11 20:45:00.540774 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.540747 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wcr4\" (UniqueName: \"kubernetes.io/projected/499ac8e2-b1d1-441b-8c03-49acfcd8f79a-kube-api-access-2wcr4\") pod \"redis-69bbd798b6-w7vsd\" (UID: \"499ac8e2-b1d1-441b-8c03-49acfcd8f79a\") " pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" Jun 11 20:45:00.674098 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.673998 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" Jun 11 20:45:00.806477 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.806371 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-w7vsd"] Jun 11 20:45:00.809089 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:45:00.809062 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod499ac8e2_b1d1_441b_8c03_49acfcd8f79a.slice/crio-f931922dbcf270b4bf30b79476404e0e7ad7819284dec203b4a4ec5712e82259 WatchSource:0}: Error finding container f931922dbcf270b4bf30b79476404e0e7ad7819284dec203b4a4ec5712e82259: Status 404 returned error can't find the container with id f931922dbcf270b4bf30b79476404e0e7ad7819284dec203b4a4ec5712e82259 Jun 11 20:45:00.820287 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:00.820256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" event={"ID":"499ac8e2-b1d1-441b-8c03-49acfcd8f79a","Type":"ContainerStarted","Data":"f931922dbcf270b4bf30b79476404e0e7ad7819284dec203b4a4ec5712e82259"} Jun 11 20:45:02.829456 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:02.829418 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" event={"ID":"499ac8e2-b1d1-441b-8c03-49acfcd8f79a","Type":"ContainerStarted","Data":"cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb"} Jun 11 20:45:02.845982 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:02.845917 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" podStartSLOduration=1.370832614 podStartE2EDuration="2.845898732s" totalImagesPullingTime="1.475066118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:45:02.842352391 +0000 UTC m=+1168.197098706" watchObservedRunningTime="2026-06-11 20:45:02.845898732 +0000 UTC m=+1168.200645047" Jun 11 20:45:05.810761 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.810728 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4xl5g"] Jun 11 20:45:05.814192 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.814169 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:45:05.816340 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.816318 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jun 11 20:45:05.818431 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.818404 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4xl5g"] Jun 11 20:45:05.877430 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.877398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8c294292-f7e6-472e-9c1c-5c7289aa464d-postgresdata\") pod \"postgres-864846d4f5-4xl5g\" (UID: \"8c294292-f7e6-472e-9c1c-5c7289aa464d\") " pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:45:05.877615 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.877440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjgpb\" (UniqueName: \"kubernetes.io/projected/8c294292-f7e6-472e-9c1c-5c7289aa464d-kube-api-access-gjgpb\") pod \"postgres-864846d4f5-4xl5g\" (UID: \"8c294292-f7e6-472e-9c1c-5c7289aa464d\") " pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:45:05.978686 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.978643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8c294292-f7e6-472e-9c1c-5c7289aa464d-postgresdata\") pod \"postgres-864846d4f5-4xl5g\" (UID: \"8c294292-f7e6-472e-9c1c-5c7289aa464d\") " pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:45:05.978686 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.978694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgpb\" (UniqueName: \"kubernetes.io/projected/8c294292-f7e6-472e-9c1c-5c7289aa464d-kube-api-access-gjgpb\") pod \"postgres-864846d4f5-4xl5g\" (UID: \"8c294292-f7e6-472e-9c1c-5c7289aa464d\") " pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:45:05.979085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.979066 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8c294292-f7e6-472e-9c1c-5c7289aa464d-postgresdata\") pod \"postgres-864846d4f5-4xl5g\" (UID: \"8c294292-f7e6-472e-9c1c-5c7289aa464d\") " pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:45:05.986255 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:05.986227 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgpb\" (UniqueName: \"kubernetes.io/projected/8c294292-f7e6-472e-9c1c-5c7289aa464d-kube-api-access-gjgpb\") pod \"postgres-864846d4f5-4xl5g\" (UID: \"8c294292-f7e6-472e-9c1c-5c7289aa464d\") " pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:45:06.126741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:06.126640 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:45:06.259629 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:06.259563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4xl5g"] Jun 11 20:45:06.262261 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:45:06.262236 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c294292_f7e6_472e_9c1c_5c7289aa464d.slice/crio-abde52ee350e11b042ce2d4bae7faa6b38ffc275c267e5f8829d8da54e729776 WatchSource:0}: Error finding container abde52ee350e11b042ce2d4bae7faa6b38ffc275c267e5f8829d8da54e729776: Status 404 returned error can't find the container with id abde52ee350e11b042ce2d4bae7faa6b38ffc275c267e5f8829d8da54e729776 Jun 11 20:45:06.845942 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:06.845906 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" event={"ID":"8c294292-f7e6-472e-9c1c-5c7289aa464d","Type":"ContainerStarted","Data":"82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6"} Jun 11 20:45:06.845942 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:06.845945 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" event={"ID":"8c294292-f7e6-472e-9c1c-5c7289aa464d","Type":"ContainerStarted","Data":"abde52ee350e11b042ce2d4bae7faa6b38ffc275c267e5f8829d8da54e729776"} Jun 11 20:45:06.860819 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:06.860771 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" podStartSLOduration=1.86075093 podStartE2EDuration="1.86075093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:45:06.860479462 +0000 UTC m=+1172.215225779" watchObservedRunningTime="2026-06-11 20:45:06.86075093 +0000 UTC m=+1172.215497246" Jun 11 20:45:11.456237 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.456199 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86"] Jun 11 20:45:11.462415 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.462393 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.464646 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.464622 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jun 11 20:45:11.464783 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.464765 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-c2dbs\"" Jun 11 20:45:11.464846 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.464783 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jun 11 20:45:11.464846 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.464789 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jun 11 20:45:11.472869 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.472843 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86"] Jun 11 20:45:11.626313 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.626275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-registry-tls\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.626313 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.626316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/550fc982-6cdc-4a4f-b74c-a6123a53b21d-feast-data\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.626545 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.626350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-online-tls\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.626545 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.626434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvctp\" (UniqueName: \"kubernetes.io/projected/550fc982-6cdc-4a4f-b74c-a6123a53b21d-kube-api-access-nvctp\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.727288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.727201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-registry-tls\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.727288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.727240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/550fc982-6cdc-4a4f-b74c-a6123a53b21d-feast-data\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.727288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.727285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-online-tls\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.727503 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.727317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvctp\" (UniqueName: \"kubernetes.io/projected/550fc982-6cdc-4a4f-b74c-a6123a53b21d-kube-api-access-nvctp\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.727645 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.727625 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/550fc982-6cdc-4a4f-b74c-a6123a53b21d-feast-data\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.729851 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.729828 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-registry-tls\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.729851 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.729842 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-online-tls\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.734303 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.734276 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvctp\" (UniqueName: \"kubernetes.io/projected/550fc982-6cdc-4a4f-b74c-a6123a53b21d-kube-api-access-nvctp\") pod \"feast-credit-scoring-56f9bfd55c-2lb86\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.773908 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.773868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:11.911654 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:11.911621 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86"] Jun 11 20:45:11.914353 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:45:11.914328 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550fc982_6cdc_4a4f_b74c_a6123a53b21d.slice/crio-7ffcc5216b70bb19e1836a6fdf6384aa8b317f7634eedf50c781dab1a243a345 WatchSource:0}: Error finding container 7ffcc5216b70bb19e1836a6fdf6384aa8b317f7634eedf50c781dab1a243a345: Status 404 returned error can't find the container with id 7ffcc5216b70bb19e1836a6fdf6384aa8b317f7634eedf50c781dab1a243a345 Jun 11 20:45:12.874413 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:12.874378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerStarted","Data":"9a3ed91f7a2e77410adea9ea05909c2ed642aa7025b81fc55599072a94c67128"} Jun 11 20:45:12.874796 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:12.874435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerStarted","Data":"7ffcc5216b70bb19e1836a6fdf6384aa8b317f7634eedf50c781dab1a243a345"} Jun 11 20:45:13.885871 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:13.885831 2564 generic.go:356] "Generic (PLEG): container finished" podID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerID="9a3ed91f7a2e77410adea9ea05909c2ed642aa7025b81fc55599072a94c67128" exitCode=0 Jun 11 20:45:13.886394 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:13.885903 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerDied","Data":"9a3ed91f7a2e77410adea9ea05909c2ed642aa7025b81fc55599072a94c67128"} Jun 11 20:45:14.891526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:14.891485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerStarted","Data":"0fbb28e9591d429cb013063355488db295c8daaad64d7a3d84be6ea171f53145"} Jun 11 20:45:21.923424 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:21.923390 2564 generic.go:356] "Generic (PLEG): container finished" podID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerID="0fbb28e9591d429cb013063355488db295c8daaad64d7a3d84be6ea171f53145" exitCode=0 Jun 11 20:45:21.923802 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:21.923463 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerDied","Data":"0fbb28e9591d429cb013063355488db295c8daaad64d7a3d84be6ea171f53145"} Jun 11 20:45:22.931305 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:22.931260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerStarted","Data":"8ce7d719f96640771addd39bcf0a3448990447189b0d0d69a34fa8190e4e9c52"} Jun 11 20:45:22.931710 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:22.931318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerStarted","Data":"3f16ebd64f87e180b8a7877a38c3a57c5c42cfc80f221e67b5dcec1b2f0479ed"} Jun 11 20:45:22.953054 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:22.952998 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podStartSLOduration=11.952978329 podStartE2EDuration="11.952978329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:45:22.951055679 +0000 UTC m=+1188.305801993" watchObservedRunningTime="2026-06-11 20:45:22.952978329 +0000 UTC m=+1188.307724637" Jun 11 20:45:23.774946 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:23.774908 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:23.775143 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:23.774988 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:23.776394 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:23.776363 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Jun 11 20:45:23.776512 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:23.776375 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Jun 11 20:45:26.774637 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:26.774588 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Jun 11 20:45:26.775040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:26.774680 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Jun 11 20:45:29.774537 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:29.774482 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Jun 11 20:45:29.780193 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:29.780162 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:29.780646 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:29.780607 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:29.783645 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:29.783622 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:31.774355 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:31.774317 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:32.774508 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:32.774450 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Jun 11 20:45:35.110815 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:35.110783 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:45:35.112319 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:35.112297 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:45:35.117264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:35.117240 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:45:35.119023 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:35.119001 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:45:35.775230 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:35.775198 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:35.775911 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:35.775895 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:45:37.283662 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.283626 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7"] Jun 11 20:45:37.289285 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.289258 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.291313 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.291249 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jun 11 20:45:37.291485 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.291314 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jun 11 20:45:37.291485 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.291446 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jun 11 20:45:37.291598 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.291500 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-54xvp\"" Jun 11 20:45:37.297328 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.297303 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7"] Jun 11 20:45:37.352582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.352540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-registry-tls\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.352772 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.352591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51adacca-6f55-4d32-8dda-3d2d014d3efa-feast-data\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.352772 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.352695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-online-tls\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.352874 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.352808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxqk\" (UniqueName: \"kubernetes.io/projected/51adacca-6f55-4d32-8dda-3d2d014d3efa-kube-api-access-txxqk\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.453524 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.453489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-online-tls\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.453717 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.453556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxqk\" (UniqueName: \"kubernetes.io/projected/51adacca-6f55-4d32-8dda-3d2d014d3efa-kube-api-access-txxqk\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.453717 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.453593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-registry-tls\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.453717 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.453608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51adacca-6f55-4d32-8dda-3d2d014d3efa-feast-data\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.454052 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.454031 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51adacca-6f55-4d32-8dda-3d2d014d3efa-feast-data\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.456183 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.456153 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-registry-tls\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.456306 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.456187 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-online-tls\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.461419 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.461393 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxqk\" (UniqueName: \"kubernetes.io/projected/51adacca-6f55-4d32-8dda-3d2d014d3efa-kube-api-access-txxqk\") pod \"feast-driver-ranking-7f5978b9c8-h4zb7\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.600757 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.600658 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:37.737262 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:37.737105 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7"] Jun 11 20:45:37.740096 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:45:37.740062 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51adacca_6f55_4d32_8dda_3d2d014d3efa.slice/crio-23a5b7bd069f61108f54fc7b0c920b330c72028cb1287d112bd2f22047634170 WatchSource:0}: Error finding container 23a5b7bd069f61108f54fc7b0c920b330c72028cb1287d112bd2f22047634170: Status 404 returned error can't find the container with id 23a5b7bd069f61108f54fc7b0c920b330c72028cb1287d112bd2f22047634170 Jun 11 20:45:38.010847 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:38.010802 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerStarted","Data":"46d4dd73528980dc086ce645e5757ff550f60e58cddb340c31339e2a3dce018c"} Jun 11 20:45:38.011074 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:38.010874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerStarted","Data":"23a5b7bd069f61108f54fc7b0c920b330c72028cb1287d112bd2f22047634170"} Jun 11 20:45:43.034655 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:43.034613 2564 generic.go:356] "Generic (PLEG): container finished" podID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerID="46d4dd73528980dc086ce645e5757ff550f60e58cddb340c31339e2a3dce018c" exitCode=0 Jun 11 20:45:43.035067 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:43.034686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerDied","Data":"46d4dd73528980dc086ce645e5757ff550f60e58cddb340c31339e2a3dce018c"} Jun 11 20:45:44.041205 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:44.041171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerStarted","Data":"44640203e60a0edd1b35a611f9d551ff8a09303e09d5990a242acf2b095fa295"} Jun 11 20:45:51.075156 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:51.075115 2564 generic.go:356] "Generic (PLEG): container finished" podID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerID="44640203e60a0edd1b35a611f9d551ff8a09303e09d5990a242acf2b095fa295" exitCode=0 Jun 11 20:45:51.077659 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:51.075128 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerDied","Data":"44640203e60a0edd1b35a611f9d551ff8a09303e09d5990a242acf2b095fa295"} Jun 11 20:45:52.083470 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:52.083430 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerStarted","Data":"1f1a6e6f9d7d028f63f43d6ff0d6a2971779ad18f95a1bedb01566e888844304"} Jun 11 20:45:52.083869 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:52.083508 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerStarted","Data":"9414698d1bd6304ccf39e6a28bf8df8e862bceaa3e36b521f1134cc4757f8e2e"} Jun 11 20:45:52.105301 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:52.105237 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podStartSLOduration=15.105213942 podStartE2EDuration="15.105213942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:45:52.103375316 +0000 UTC m=+1217.458121633" watchObservedRunningTime="2026-06-11 20:45:52.105213942 +0000 UTC m=+1217.459960259" Jun 11 20:45:52.601877 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:52.601834 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:52.602088 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:52.601914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:52.603708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:52.603677 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" Jun 11 20:45:52.603836 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:52.603746 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Jun 11 20:45:55.601677 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:55.601622 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" Jun 11 20:45:55.602167 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:55.601710 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Jun 11 20:45:57.600786 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:57.600742 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:57.601301 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:57.600810 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:58.601532 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:58.601476 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" Jun 11 20:45:58.606795 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:58.606760 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:45:58.610353 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:45:58.610327 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:46:01.601727 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:46:01.601679 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" Jun 11 20:46:04.601484 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:46:04.601455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:46:04.602198 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:46:04.602178 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:47:23.078854 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.078823 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4"] Jun 11 20:47:23.079339 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.078999 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" podUID="df5524a3-60eb-413e-bc6e-4d50f3b14744" containerName="manager" containerID="cri-o://7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626" gracePeriod=10 Jun 11 20:47:23.318460 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.318425 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86"] Jun 11 20:47:23.318736 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.318712 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="registry" containerID="cri-o://3f16ebd64f87e180b8a7877a38c3a57c5c42cfc80f221e67b5dcec1b2f0479ed" gracePeriod=30 Jun 11 20:47:23.319010 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.318830 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="online" containerID="cri-o://8ce7d719f96640771addd39bcf0a3448990447189b0d0d69a34fa8190e4e9c52" gracePeriod=30 Jun 11 20:47:23.320549 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.320526 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7"] Jun 11 20:47:23.320984 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.320944 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="registry" containerID="cri-o://9414698d1bd6304ccf39e6a28bf8df8e862bceaa3e36b521f1134cc4757f8e2e" gracePeriod=30 Jun 11 20:47:23.321279 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.321013 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="online" containerID="cri-o://1f1a6e6f9d7d028f63f43d6ff0d6a2971779ad18f95a1bedb01566e888844304" gracePeriod=30 Jun 11 20:47:23.373370 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.373343 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" Jun 11 20:47:23.487851 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.487816 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f96kx\" (UniqueName: \"kubernetes.io/projected/df5524a3-60eb-413e-bc6e-4d50f3b14744-kube-api-access-f96kx\") pod \"df5524a3-60eb-413e-bc6e-4d50f3b14744\" (UID: \"df5524a3-60eb-413e-bc6e-4d50f3b14744\") " Jun 11 20:47:23.490055 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.490021 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5524a3-60eb-413e-bc6e-4d50f3b14744-kube-api-access-f96kx" (OuterVolumeSpecName: "kube-api-access-f96kx") pod "df5524a3-60eb-413e-bc6e-4d50f3b14744" (UID: "df5524a3-60eb-413e-bc6e-4d50f3b14744"). InnerVolumeSpecName "kube-api-access-f96kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:47:23.502006 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.501958 2564 generic.go:356] "Generic (PLEG): container finished" podID="df5524a3-60eb-413e-bc6e-4d50f3b14744" containerID="7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626" exitCode=0 Jun 11 20:47:23.502154 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.502031 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" Jun 11 20:47:23.502154 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.502041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" event={"ID":"df5524a3-60eb-413e-bc6e-4d50f3b14744","Type":"ContainerDied","Data":"7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626"} Jun 11 20:47:23.502154 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.502113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4" event={"ID":"df5524a3-60eb-413e-bc6e-4d50f3b14744","Type":"ContainerDied","Data":"d78b2048d3bf4e1a59ae6d2c99f4aa252a61a178626522b5fe5f9a5cef6bb412"} Jun 11 20:47:23.502154 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.502136 2564 scope.go:117] "RemoveContainer" containerID="7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626" Jun 11 20:47:23.511037 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.511017 2564 scope.go:117] "RemoveContainer" containerID="7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626" Jun 11 20:47:23.511328 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:47:23.511305 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626\": container with ID starting with 7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626 not found: ID does not exist" containerID="7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626" Jun 11 20:47:23.511390 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.511341 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626"} err="failed to get container status \"7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626\": rpc error: code = NotFound desc = could not find container \"7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626\": container with ID starting with 7cb38df4368990cb71abf405c2b6bb6357bfbace3d870e3e11bda8ddcb1f6626 not found: ID does not exist" Jun 11 20:47:23.522719 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.522692 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4"] Jun 11 20:47:23.524995 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.524955 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5b4f6ff995-ck9f4"] Jun 11 20:47:23.589078 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:23.589038 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96kx\" (UniqueName: \"kubernetes.io/projected/df5524a3-60eb-413e-bc6e-4d50f3b14744-kube-api-access-f96kx\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:24.509009 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:24.508950 2564 generic.go:356] "Generic (PLEG): container finished" podID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerID="1f1a6e6f9d7d028f63f43d6ff0d6a2971779ad18f95a1bedb01566e888844304" exitCode=0 Jun 11 20:47:24.509470 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:24.509024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerDied","Data":"1f1a6e6f9d7d028f63f43d6ff0d6a2971779ad18f95a1bedb01566e888844304"} Jun 11 20:47:24.512348 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:24.512324 2564 generic.go:356] "Generic (PLEG): container finished" podID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerID="8ce7d719f96640771addd39bcf0a3448990447189b0d0d69a34fa8190e4e9c52" exitCode=0 Jun 11 20:47:24.512462 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:24.512380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerDied","Data":"8ce7d719f96640771addd39bcf0a3448990447189b0d0d69a34fa8190e4e9c52"} Jun 11 20:47:25.075699 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.075654 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df5524a3-60eb-413e-bc6e-4d50f3b14744" path="/var/lib/kubelet/pods/df5524a3-60eb-413e-bc6e-4d50f3b14744/volumes" Jun 11 20:47:25.476480 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.476395 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4xl5g"] Jun 11 20:47:25.476637 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.476564 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" podUID="8c294292-f7e6-472e-9c1c-5c7289aa464d" containerName="postgres" containerID="cri-o://82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6" gracePeriod=30 Jun 11 20:47:25.481059 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.481032 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-w7vsd"] Jun 11 20:47:25.481239 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.481206 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" podUID="499ac8e2-b1d1-441b-8c03-49acfcd8f79a" containerName="redis" containerID="cri-o://cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb" gracePeriod=30 Jun 11 20:47:25.743013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.742991 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:47:25.746010 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.745984 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" Jun 11 20:47:25.807154 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.807124 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjgpb\" (UniqueName: \"kubernetes.io/projected/8c294292-f7e6-472e-9c1c-5c7289aa464d-kube-api-access-gjgpb\") pod \"8c294292-f7e6-472e-9c1c-5c7289aa464d\" (UID: \"8c294292-f7e6-472e-9c1c-5c7289aa464d\") " Jun 11 20:47:25.807349 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.807167 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8c294292-f7e6-472e-9c1c-5c7289aa464d-postgresdata\") pod \"8c294292-f7e6-472e-9c1c-5c7289aa464d\" (UID: \"8c294292-f7e6-472e-9c1c-5c7289aa464d\") " Jun 11 20:47:25.807349 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.807222 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wcr4\" (UniqueName: \"kubernetes.io/projected/499ac8e2-b1d1-441b-8c03-49acfcd8f79a-kube-api-access-2wcr4\") pod \"499ac8e2-b1d1-441b-8c03-49acfcd8f79a\" (UID: \"499ac8e2-b1d1-441b-8c03-49acfcd8f79a\") " Jun 11 20:47:25.809357 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.809319 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c294292-f7e6-472e-9c1c-5c7289aa464d-kube-api-access-gjgpb" (OuterVolumeSpecName: "kube-api-access-gjgpb") pod "8c294292-f7e6-472e-9c1c-5c7289aa464d" (UID: "8c294292-f7e6-472e-9c1c-5c7289aa464d"). InnerVolumeSpecName "kube-api-access-gjgpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:47:25.809478 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.809332 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499ac8e2-b1d1-441b-8c03-49acfcd8f79a-kube-api-access-2wcr4" (OuterVolumeSpecName: "kube-api-access-2wcr4") pod "499ac8e2-b1d1-441b-8c03-49acfcd8f79a" (UID: "499ac8e2-b1d1-441b-8c03-49acfcd8f79a"). InnerVolumeSpecName "kube-api-access-2wcr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:47:25.837681 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.837646 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c294292-f7e6-472e-9c1c-5c7289aa464d-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "8c294292-f7e6-472e-9c1c-5c7289aa464d" (UID: "8c294292-f7e6-472e-9c1c-5c7289aa464d"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:47:25.908557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.908495 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjgpb\" (UniqueName: \"kubernetes.io/projected/8c294292-f7e6-472e-9c1c-5c7289aa464d-kube-api-access-gjgpb\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:25.908557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.908544 2564 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8c294292-f7e6-472e-9c1c-5c7289aa464d-postgresdata\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:25.908557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:25.908555 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wcr4\" (UniqueName: \"kubernetes.io/projected/499ac8e2-b1d1-441b-8c03-49acfcd8f79a-kube-api-access-2wcr4\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:26.523065 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.523025 2564 generic.go:356] "Generic (PLEG): container finished" podID="499ac8e2-b1d1-441b-8c03-49acfcd8f79a" containerID="cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb" exitCode=0 Jun 11 20:47:26.523065 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.523066 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" Jun 11 20:47:26.523307 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.523102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" event={"ID":"499ac8e2-b1d1-441b-8c03-49acfcd8f79a","Type":"ContainerDied","Data":"cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb"} Jun 11 20:47:26.523307 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.523138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-w7vsd" event={"ID":"499ac8e2-b1d1-441b-8c03-49acfcd8f79a","Type":"ContainerDied","Data":"f931922dbcf270b4bf30b79476404e0e7ad7819284dec203b4a4ec5712e82259"} Jun 11 20:47:26.523307 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.523154 2564 scope.go:117] "RemoveContainer" containerID="cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb" Jun 11 20:47:26.524557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.524458 2564 generic.go:356] "Generic (PLEG): container finished" podID="8c294292-f7e6-472e-9c1c-5c7289aa464d" containerID="82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6" exitCode=0 Jun 11 20:47:26.524557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.524495 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" Jun 11 20:47:26.524557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.524517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" event={"ID":"8c294292-f7e6-472e-9c1c-5c7289aa464d","Type":"ContainerDied","Data":"82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6"} Jun 11 20:47:26.524557 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.524542 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4xl5g" event={"ID":"8c294292-f7e6-472e-9c1c-5c7289aa464d","Type":"ContainerDied","Data":"abde52ee350e11b042ce2d4bae7faa6b38ffc275c267e5f8829d8da54e729776"} Jun 11 20:47:26.533554 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.533529 2564 scope.go:117] "RemoveContainer" containerID="cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb" Jun 11 20:47:26.533860 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:47:26.533839 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb\": container with ID starting with cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb not found: ID does not exist" containerID="cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb" Jun 11 20:47:26.533948 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.533870 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb"} err="failed to get container status \"cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb\": rpc error: code = NotFound desc = could not find container \"cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb\": container with ID starting with cb7d7e7b24edc3570150c5ddc81b958cffd9fdba85a3d633d6956eec0d5d4dfb not found: ID does not exist" Jun 11 20:47:26.533948 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.533886 2564 scope.go:117] "RemoveContainer" containerID="82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6" Jun 11 20:47:26.541804 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.541781 2564 scope.go:117] "RemoveContainer" containerID="82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6" Jun 11 20:47:26.542088 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:47:26.542058 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6\": container with ID starting with 82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6 not found: ID does not exist" containerID="82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6" Jun 11 20:47:26.542171 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.542101 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6"} err="failed to get container status \"82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6\": rpc error: code = NotFound desc = could not find container \"82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6\": container with ID starting with 82bc5fc902c92e20fcadc9645611ae3b8af8bc7dae318188a623e368c5fe59c6 not found: ID does not exist" Jun 11 20:47:26.545576 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.545556 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-w7vsd"] Jun 11 20:47:26.549268 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.549245 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-w7vsd"] Jun 11 20:47:26.558425 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.558391 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4xl5g"] Jun 11 20:47:26.559900 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:26.559880 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4xl5g"] Jun 11 20:47:27.075166 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:27.075135 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="499ac8e2-b1d1-441b-8c03-49acfcd8f79a" path="/var/lib/kubelet/pods/499ac8e2-b1d1-441b-8c03-49acfcd8f79a/volumes" Jun 11 20:47:27.075521 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:27.075492 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c294292-f7e6-472e-9c1c-5c7289aa464d" path="/var/lib/kubelet/pods/8c294292-f7e6-472e-9c1c-5c7289aa464d/volumes" Jun 11 20:47:28.607611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:28.607563 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Jun 11 20:47:29.781849 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:29.781801 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Jun 11 20:47:38.608329 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:38.608278 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Jun 11 20:47:39.781349 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:39.781303 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Jun 11 20:47:48.608064 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:48.608010 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Jun 11 20:47:48.608460 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:48.608163 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:47:49.781509 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:49.781468 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Jun 11 20:47:49.781941 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:49.781607 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:47:53.640743 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:53.640649 2564 generic.go:356] "Generic (PLEG): container finished" podID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerID="9414698d1bd6304ccf39e6a28bf8df8e862bceaa3e36b521f1134cc4757f8e2e" exitCode=137 Jun 11 20:47:53.640743 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:53.640707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerDied","Data":"9414698d1bd6304ccf39e6a28bf8df8e862bceaa3e36b521f1134cc4757f8e2e"} Jun 11 20:47:53.642959 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:53.642936 2564 generic.go:356] "Generic (PLEG): container finished" podID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerID="3f16ebd64f87e180b8a7877a38c3a57c5c42cfc80f221e67b5dcec1b2f0479ed" exitCode=137 Jun 11 20:47:53.643097 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:53.642991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerDied","Data":"3f16ebd64f87e180b8a7877a38c3a57c5c42cfc80f221e67b5dcec1b2f0479ed"} Jun 11 20:47:54.000096 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.000068 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:47:54.002514 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.002489 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:47:54.155773 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.155734 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51adacca-6f55-4d32-8dda-3d2d014d3efa-feast-data\") pod \"51adacca-6f55-4d32-8dda-3d2d014d3efa\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " Jun 11 20:47:54.155999 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.155802 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-registry-tls\") pod \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " Jun 11 20:47:54.155999 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.155852 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-registry-tls\") pod \"51adacca-6f55-4d32-8dda-3d2d014d3efa\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " Jun 11 20:47:54.155999 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.155885 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvctp\" (UniqueName: \"kubernetes.io/projected/550fc982-6cdc-4a4f-b74c-a6123a53b21d-kube-api-access-nvctp\") pod \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " Jun 11 20:47:54.155999 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.155912 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-online-tls\") pod \"51adacca-6f55-4d32-8dda-3d2d014d3efa\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " Jun 11 20:47:54.155999 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.155935 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/550fc982-6cdc-4a4f-b74c-a6123a53b21d-feast-data\") pod \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " Jun 11 20:47:54.156698 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.156347 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txxqk\" (UniqueName: \"kubernetes.io/projected/51adacca-6f55-4d32-8dda-3d2d014d3efa-kube-api-access-txxqk\") pod \"51adacca-6f55-4d32-8dda-3d2d014d3efa\" (UID: \"51adacca-6f55-4d32-8dda-3d2d014d3efa\") " Jun 11 20:47:54.156698 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.156394 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-online-tls\") pod \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\" (UID: \"550fc982-6cdc-4a4f-b74c-a6123a53b21d\") " Jun 11 20:47:54.156698 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.156435 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51adacca-6f55-4d32-8dda-3d2d014d3efa-feast-data" (OuterVolumeSpecName: "feast-data") pod "51adacca-6f55-4d32-8dda-3d2d014d3efa" (UID: "51adacca-6f55-4d32-8dda-3d2d014d3efa"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:47:54.156698 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.156664 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/51adacca-6f55-4d32-8dda-3d2d014d3efa-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:54.158546 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.158427 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-online-tls" (OuterVolumeSpecName: "online-tls") pod "51adacca-6f55-4d32-8dda-3d2d014d3efa" (UID: "51adacca-6f55-4d32-8dda-3d2d014d3efa"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:47:54.158546 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.158582 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "51adacca-6f55-4d32-8dda-3d2d014d3efa" (UID: "51adacca-6f55-4d32-8dda-3d2d014d3efa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:47:54.158546 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.158591 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550fc982-6cdc-4a4f-b74c-a6123a53b21d-kube-api-access-nvctp" (OuterVolumeSpecName: "kube-api-access-nvctp") pod "550fc982-6cdc-4a4f-b74c-a6123a53b21d" (UID: "550fc982-6cdc-4a4f-b74c-a6123a53b21d"). InnerVolumeSpecName "kube-api-access-nvctp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:47:54.158546 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.158702 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51adacca-6f55-4d32-8dda-3d2d014d3efa-kube-api-access-txxqk" (OuterVolumeSpecName: "kube-api-access-txxqk") pod "51adacca-6f55-4d32-8dda-3d2d014d3efa" (UID: "51adacca-6f55-4d32-8dda-3d2d014d3efa"). InnerVolumeSpecName "kube-api-access-txxqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:47:54.159068 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.158758 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "550fc982-6cdc-4a4f-b74c-a6123a53b21d" (UID: "550fc982-6cdc-4a4f-b74c-a6123a53b21d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:47:54.159068 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.158864 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-online-tls" (OuterVolumeSpecName: "online-tls") pod "550fc982-6cdc-4a4f-b74c-a6123a53b21d" (UID: "550fc982-6cdc-4a4f-b74c-a6123a53b21d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:47:54.164862 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.164828 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550fc982-6cdc-4a4f-b74c-a6123a53b21d-feast-data" (OuterVolumeSpecName: "feast-data") pod "550fc982-6cdc-4a4f-b74c-a6123a53b21d" (UID: "550fc982-6cdc-4a4f-b74c-a6123a53b21d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:47:54.258095 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.258055 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:54.258095 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.258096 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvctp\" (UniqueName: \"kubernetes.io/projected/550fc982-6cdc-4a4f-b74c-a6123a53b21d-kube-api-access-nvctp\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:54.258296 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.258125 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/51adacca-6f55-4d32-8dda-3d2d014d3efa-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:54.258296 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.258139 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/550fc982-6cdc-4a4f-b74c-a6123a53b21d-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:54.258296 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.258149 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txxqk\" (UniqueName: \"kubernetes.io/projected/51adacca-6f55-4d32-8dda-3d2d014d3efa-kube-api-access-txxqk\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:54.258296 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.258157 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:54.258296 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.258165 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/550fc982-6cdc-4a4f-b74c-a6123a53b21d-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:47:54.649741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.649648 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" event={"ID":"51adacca-6f55-4d32-8dda-3d2d014d3efa","Type":"ContainerDied","Data":"23a5b7bd069f61108f54fc7b0c920b330c72028cb1287d112bd2f22047634170"} Jun 11 20:47:54.649741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.649655 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7" Jun 11 20:47:54.649741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.649713 2564 scope.go:117] "RemoveContainer" containerID="1f1a6e6f9d7d028f63f43d6ff0d6a2971779ad18f95a1bedb01566e888844304" Jun 11 20:47:54.652259 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.652236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" event={"ID":"550fc982-6cdc-4a4f-b74c-a6123a53b21d","Type":"ContainerDied","Data":"7ffcc5216b70bb19e1836a6fdf6384aa8b317f7634eedf50c781dab1a243a345"} Jun 11 20:47:54.652372 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.652268 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86" Jun 11 20:47:54.663911 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.663774 2564 scope.go:117] "RemoveContainer" containerID="9414698d1bd6304ccf39e6a28bf8df8e862bceaa3e36b521f1134cc4757f8e2e" Jun 11 20:47:54.672464 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.672446 2564 scope.go:117] "RemoveContainer" containerID="44640203e60a0edd1b35a611f9d551ff8a09303e09d5990a242acf2b095fa295" Jun 11 20:47:54.678443 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.678411 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7"] Jun 11 20:47:54.680360 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.680325 2564 scope.go:117] "RemoveContainer" containerID="46d4dd73528980dc086ce645e5757ff550f60e58cddb340c31339e2a3dce018c" Jun 11 20:47:54.681797 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.681777 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f5978b9c8-h4zb7"] Jun 11 20:47:54.688020 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.688003 2564 scope.go:117] "RemoveContainer" containerID="8ce7d719f96640771addd39bcf0a3448990447189b0d0d69a34fa8190e4e9c52" Jun 11 20:47:54.691018 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.690996 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86"] Jun 11 20:47:54.694909 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.694885 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f9bfd55c-2lb86"] Jun 11 20:47:54.696057 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.696038 2564 scope.go:117] "RemoveContainer" containerID="3f16ebd64f87e180b8a7877a38c3a57c5c42cfc80f221e67b5dcec1b2f0479ed" Jun 11 20:47:54.703821 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.703802 2564 scope.go:117] "RemoveContainer" containerID="0fbb28e9591d429cb013063355488db295c8daaad64d7a3d84be6ea171f53145" Jun 11 20:47:54.710865 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:54.710847 2564 scope.go:117] "RemoveContainer" containerID="9a3ed91f7a2e77410adea9ea05909c2ed642aa7025b81fc55599072a94c67128" Jun 11 20:47:55.074986 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:55.074938 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" path="/var/lib/kubelet/pods/51adacca-6f55-4d32-8dda-3d2d014d3efa/volumes" Jun 11 20:47:55.075468 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:47:55.075455 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" path="/var/lib/kubelet/pods/550fc982-6cdc-4a4f-b74c-a6123a53b21d/volumes" Jun 11 20:48:01.816511 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.816463 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s"] Jun 11 20:48:01.816904 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.816889 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5524a3-60eb-413e-bc6e-4d50f3b14744" containerName="manager" Jun 11 20:48:01.816959 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.816906 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5524a3-60eb-413e-bc6e-4d50f3b14744" containerName="manager" Jun 11 20:48:01.816959 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.816919 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c294292-f7e6-472e-9c1c-5c7289aa464d" containerName="postgres" Jun 11 20:48:01.816959 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.816927 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c294292-f7e6-472e-9c1c-5c7289aa464d" containerName="postgres" Jun 11 20:48:01.816959 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.816946 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="online" Jun 11 20:48:01.816959 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.816954 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="online" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.816979 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="registry" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.816988 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="registry" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817006 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="499ac8e2-b1d1-441b-8c03-49acfcd8f79a" containerName="redis" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817014 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="499ac8e2-b1d1-441b-8c03-49acfcd8f79a" containerName="redis" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817023 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="feast-apply" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817031 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="feast-apply" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817042 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="feast-apply" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817050 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="feast-apply" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817059 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="registry" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817067 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="registry" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817079 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="feast-init" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817087 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="feast-init" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817100 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="feast-init" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817109 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="feast-init" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817116 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="online" Jun 11 20:48:01.817144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817124 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="online" Jun 11 20:48:01.817603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817190 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c294292-f7e6-472e-9c1c-5c7289aa464d" containerName="postgres" Jun 11 20:48:01.817603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817203 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="499ac8e2-b1d1-441b-8c03-49acfcd8f79a" containerName="redis" Jun 11 20:48:01.817603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817212 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="registry" Jun 11 20:48:01.817603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817221 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="online" Jun 11 20:48:01.817603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817231 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="550fc982-6cdc-4a4f-b74c-a6123a53b21d" containerName="online" Jun 11 20:48:01.817603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817238 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="51adacca-6f55-4d32-8dda-3d2d014d3efa" containerName="registry" Jun 11 20:48:01.817603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.817247 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5524a3-60eb-413e-bc6e-4d50f3b14744" containerName="manager" Jun 11 20:48:01.821085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.821066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:01.823372 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.823349 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 11 20:48:01.823577 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.823562 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 11 20:48:01.823699 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.823613 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 11 20:48:01.823770 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.823718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 11 20:48:01.823770 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.823744 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 11 20:48:01.823893 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.823832 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 11 20:48:01.823958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.823946 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-jxn58\"" Jun 11 20:48:01.831911 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.831884 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s"] Jun 11 20:48:01.917395 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.917358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:01.917581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.917410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/959366ef-3a07-43f5-bf5b-3d62797c6f35-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:01.917581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.917472 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:01.917581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.917501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7zlf\" (UniqueName: \"kubernetes.io/projected/959366ef-3a07-43f5-bf5b-3d62797c6f35-kube-api-access-w7zlf\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:01.917581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.917526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:01.917581 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:01.917552 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.018776 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.018740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.019013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.018789 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/959366ef-3a07-43f5-bf5b-3d62797c6f35-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.019013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.018825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.019013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.018844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7zlf\" (UniqueName: \"kubernetes.io/projected/959366ef-3a07-43f5-bf5b-3d62797c6f35-kube-api-access-w7zlf\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.019013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.018862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.019013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.018887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.019013 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:48:02.018995 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:48:02.019368 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:48:02.019080 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-ui-tls podName:959366ef-3a07-43f5-bf5b-3d62797c6f35 nodeName:}" failed. No retries permitted until 2026-06-11 20:48:02.51905718 +0000 UTC m=+1347.873803475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-ui-tls") pod "feast-simple-feast-setup-75c64c888b-rsx5s" (UID: "959366ef-3a07-43f5-bf5b-3d62797c6f35") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:48:02.019440 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.019411 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/959366ef-3a07-43f5-bf5b-3d62797c6f35-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.021387 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.021357 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.021487 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.021459 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.021598 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.021583 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.026666 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.026644 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7zlf\" (UniqueName: \"kubernetes.io/projected/959366ef-3a07-43f5-bf5b-3d62797c6f35-kube-api-access-w7zlf\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.523044 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.523002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.525397 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.525373 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-rsx5s\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.743238 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.743203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:02.880072 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.880038 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s"] Jun 11 20:48:02.882905 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:48:02.882866 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod959366ef_3a07_43f5_bf5b_3d62797c6f35.slice/crio-4044fc3c904885c438297c022b50f5415353036c48cf68128189ab194a401dae WatchSource:0}: Error finding container 4044fc3c904885c438297c022b50f5415353036c48cf68128189ab194a401dae: Status 404 returned error can't find the container with id 4044fc3c904885c438297c022b50f5415353036c48cf68128189ab194a401dae Jun 11 20:48:02.884998 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:02.884957 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 11 20:48:03.700411 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:03.700361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerStarted","Data":"4044fc3c904885c438297c022b50f5415353036c48cf68128189ab194a401dae"} Jun 11 20:48:18.773258 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:18.773218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerStarted","Data":"70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6"} Jun 11 20:48:23.793127 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:23.793098 2564 generic.go:356] "Generic (PLEG): container finished" podID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerID="70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6" exitCode=0 Jun 11 20:48:23.793623 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:23.793170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerDied","Data":"70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6"} Jun 11 20:48:24.798837 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:24.798796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerStarted","Data":"cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7"} Jun 11 20:48:29.821015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:29.820906 2564 generic.go:356] "Generic (PLEG): container finished" podID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerID="cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7" exitCode=0 Jun 11 20:48:29.821015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:29.821000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerDied","Data":"cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7"} Jun 11 20:48:30.831226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:30.831186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerStarted","Data":"0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe"} Jun 11 20:48:30.831731 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:30.831250 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerStarted","Data":"ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602"} Jun 11 20:48:30.831731 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:30.831271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerStarted","Data":"8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3"} Jun 11 20:48:30.831731 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:30.831288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerStarted","Data":"89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4"} Jun 11 20:48:30.855484 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:30.855414 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podStartSLOduration=14.140736055 podStartE2EDuration="29.855393033s" totalImagesPullingTime="15.714656978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:48:30.853580957 +0000 UTC m=+1376.208327320" watchObservedRunningTime="2026-06-11 20:48:30.855393033 +0000 UTC m=+1376.210139352" Jun 11 20:48:32.743669 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.743631 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:32.744269 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.744245 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:32.744269 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.744276 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:32.744577 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.744294 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:32.744577 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.744315 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:32.744577 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.744331 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:32.744577 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.744348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:32.744577 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.744364 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:32.746047 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.746014 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" Jun 11 20:48:32.746172 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.746066 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.35:8816: connect: connection refused" Jun 11 20:48:32.746172 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.746146 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Jun 11 20:48:32.746288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:32.746026 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.35:6571: connect: connection refused" Jun 11 20:48:35.745037 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:35.745001 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:35.745489 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:35.745329 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:35.745489 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:35.745451 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:35.746143 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:35.746113 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:35.746279 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:35.746191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:35.746279 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:35.746232 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:35.749646 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:35.749620 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:48:35.862221 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:48:35.862191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:49:38.453949 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.453856 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s"] Jun 11 20:49:38.454503 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.454190 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="registry" containerID="cri-o://89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4" gracePeriod=30 Jun 11 20:49:38.454503 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.454350 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="offline" containerID="cri-o://ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602" gracePeriod=30 Jun 11 20:49:38.454503 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.454360 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="online" containerID="cri-o://8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3" gracePeriod=30 Jun 11 20:49:38.454702 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.454460 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="ui" containerID="cri-o://0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe" gracePeriod=30 Jun 11 20:49:38.726350 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.726250 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc"] Jun 11 20:49:38.730804 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.730780 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.732958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.732930 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-z8vg4\"" Jun 11 20:49:38.744127 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.742894 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc"] Jun 11 20:49:38.767726 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.767663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.767905 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.767780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rvx\" (UniqueName: \"kubernetes.io/projected/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-kube-api-access-79rvx\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.767905 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.767831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.767905 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.767874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.768112 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.767987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.768112 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.768012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.869372 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.869324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.869372 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.869375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.869649 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.869407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.869649 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.869462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rvx\" (UniqueName: \"kubernetes.io/projected/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-kube-api-access-79rvx\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.869649 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.869495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.869649 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.869535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.869649 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:49:38.869551 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:49:38.869649 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:49:38.869646 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:49:39.369621467 +0000 UTC m=+1444.724367782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:49:38.870002 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.869985 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.870245 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:49:38.870227 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:49:38.870303 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:49:38.870288 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:49:39.370270272 +0000 UTC m=+1444.725016594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:49:38.870406 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:49:38.870374 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:49:38.870471 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:49:38.870448 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:49:39.370433286 +0000 UTC m=+1444.725179590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:49:38.872975 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.872946 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:38.878715 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:38.878667 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rvx\" (UniqueName: \"kubernetes.io/projected/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-kube-api-access-79rvx\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:39.113680 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:39.113643 2564 generic.go:356] "Generic (PLEG): container finished" podID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerID="0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe" exitCode=0 Jun 11 20:49:39.113680 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:39.113665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerDied","Data":"0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe"} Jun 11 20:49:39.374188 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:39.374147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:39.374188 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:39.374192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:39.374413 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:39.374220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:39.374413 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:49:39.374310 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:49:39.374413 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:49:39.374375 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:49:40.37435797 +0000 UTC m=+1445.729104265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:49:39.376682 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:39.376657 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:39.376807 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:39.376663 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:40.120478 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:40.120445 2564 generic.go:356] "Generic (PLEG): container finished" podID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerID="8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3" exitCode=0 Jun 11 20:49:40.120860 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:40.120484 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerDied","Data":"8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3"} Jun 11 20:49:40.382894 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:40.382796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:40.385217 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:40.385188 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-mw7gc\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:40.545679 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:40.545640 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:40.688002 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:40.687957 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc"] Jun 11 20:49:40.690757 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:49:40.690725 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod025cce3d_b61e_41a1_b58e_1c7fcbb636b6.slice/crio-c7aafaf50ed8450bf354d53750f9711e6025e5b668a7ee79954aa1fd251dfb21 WatchSource:0}: Error finding container c7aafaf50ed8450bf354d53750f9711e6025e5b668a7ee79954aa1fd251dfb21: Status 404 returned error can't find the container with id c7aafaf50ed8450bf354d53750f9711e6025e5b668a7ee79954aa1fd251dfb21 Jun 11 20:49:41.126928 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:41.126893 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerStarted","Data":"4981a4c89488f27de252ed1b5d196be1af14ac1e8e870913e579cca8df039cf1"} Jun 11 20:49:41.127336 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:41.126937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerStarted","Data":"c7aafaf50ed8450bf354d53750f9711e6025e5b668a7ee79954aa1fd251dfb21"} Jun 11 20:49:45.746678 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:45.746613 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" Jun 11 20:49:45.859620 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:45.859508 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Jun 11 20:49:46.150887 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:46.150794 2564 generic.go:356] "Generic (PLEG): container finished" podID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerID="4981a4c89488f27de252ed1b5d196be1af14ac1e8e870913e579cca8df039cf1" exitCode=0 Jun 11 20:49:46.150887 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:46.150868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerDied","Data":"4981a4c89488f27de252ed1b5d196be1af14ac1e8e870913e579cca8df039cf1"} Jun 11 20:49:47.156505 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:47.156458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerStarted","Data":"b977815f2e5c843df78d30cf18e886786d386d851e24472e6a6755b3290cd8e8"} Jun 11 20:49:52.179416 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:52.179320 2564 generic.go:356] "Generic (PLEG): container finished" podID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerID="b977815f2e5c843df78d30cf18e886786d386d851e24472e6a6755b3290cd8e8" exitCode=0 Jun 11 20:49:52.179416 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:52.179395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerDied","Data":"b977815f2e5c843df78d30cf18e886786d386d851e24472e6a6755b3290cd8e8"} Jun 11 20:49:53.190211 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:53.190168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerStarted","Data":"654bfcbedcfff96baf932bc3014b1e2261ab7639d0f7e49e829431067ff98df6"} Jun 11 20:49:53.190718 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:53.190237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerStarted","Data":"3d97104e5a5f472174c6a7d7b0bf684620088ce51df42e3bfc102acad63b7eeb"} Jun 11 20:49:53.190718 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:53.190258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerStarted","Data":"50c206469b661fe7f2ae6cbb6f58dd975e848df75014088ea091689cc8936b56"} Jun 11 20:49:53.190718 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:53.190277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerStarted","Data":"d3a1e3ad907bfcf8b67d9af1825df7a4f78bc283d2024e00b63860b537f89deb"} Jun 11 20:49:53.216321 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:53.216241 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podStartSLOduration=15.216219989 podStartE2EDuration="15.216219989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:49:53.212585814 +0000 UTC m=+1458.567332135" watchObservedRunningTime="2026-06-11 20:49:53.216219989 +0000 UTC m=+1458.570966307" Jun 11 20:49:55.546559 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.546436 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:55.547063 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.546602 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:55.547063 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.546624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:55.547477 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.547353 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:55.548812 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.548696 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.36:8816: connect: connection refused" Jun 11 20:49:55.548812 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.548714 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Jun 11 20:49:55.548812 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.548742 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jun 11 20:49:55.549106 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.548837 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" Jun 11 20:49:55.746493 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.746429 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" Jun 11 20:49:55.859624 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:55.859506 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Jun 11 20:49:58.547026 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.546988 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:58.547443 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.547189 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:58.547504 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.547449 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:58.547560 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.547524 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:58.547615 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.547565 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:58.547615 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.547587 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:58.547879 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.547857 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:58.548004 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.547914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:58.548104 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.548088 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:58.551686 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:58.551665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:59.227119 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:59.227085 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:49:59.230124 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:49:59.230105 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:50:05.746452 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:05.746400 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" Jun 11 20:50:05.746919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:05.746552 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:50:05.859984 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:05.859931 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Jun 11 20:50:05.860167 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:05.860079 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:50:09.100851 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.100824 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:50:09.260861 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.260826 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-registry-tls\") pod \"959366ef-3a07-43f5-bf5b-3d62797c6f35\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " Jun 11 20:50:09.261091 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.260892 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-ui-tls\") pod \"959366ef-3a07-43f5-bf5b-3d62797c6f35\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " Jun 11 20:50:09.261091 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.260941 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-online-tls\") pod \"959366ef-3a07-43f5-bf5b-3d62797c6f35\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " Jun 11 20:50:09.261091 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.261071 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/959366ef-3a07-43f5-bf5b-3d62797c6f35-feast-data\") pod \"959366ef-3a07-43f5-bf5b-3d62797c6f35\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " Jun 11 20:50:09.261288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.261103 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7zlf\" (UniqueName: \"kubernetes.io/projected/959366ef-3a07-43f5-bf5b-3d62797c6f35-kube-api-access-w7zlf\") pod \"959366ef-3a07-43f5-bf5b-3d62797c6f35\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " Jun 11 20:50:09.261288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.261147 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-offline-tls\") pod \"959366ef-3a07-43f5-bf5b-3d62797c6f35\" (UID: \"959366ef-3a07-43f5-bf5b-3d62797c6f35\") " Jun 11 20:50:09.261684 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.261626 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959366ef-3a07-43f5-bf5b-3d62797c6f35-feast-data" (OuterVolumeSpecName: "feast-data") pod "959366ef-3a07-43f5-bf5b-3d62797c6f35" (UID: "959366ef-3a07-43f5-bf5b-3d62797c6f35"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:50:09.263335 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.263309 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "959366ef-3a07-43f5-bf5b-3d62797c6f35" (UID: "959366ef-3a07-43f5-bf5b-3d62797c6f35"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:50:09.263449 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.263328 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "959366ef-3a07-43f5-bf5b-3d62797c6f35" (UID: "959366ef-3a07-43f5-bf5b-3d62797c6f35"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:50:09.263449 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.263401 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "959366ef-3a07-43f5-bf5b-3d62797c6f35" (UID: "959366ef-3a07-43f5-bf5b-3d62797c6f35"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:50:09.263554 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.263490 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-online-tls" (OuterVolumeSpecName: "online-tls") pod "959366ef-3a07-43f5-bf5b-3d62797c6f35" (UID: "959366ef-3a07-43f5-bf5b-3d62797c6f35"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:50:09.263554 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.263508 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959366ef-3a07-43f5-bf5b-3d62797c6f35-kube-api-access-w7zlf" (OuterVolumeSpecName: "kube-api-access-w7zlf") pod "959366ef-3a07-43f5-bf5b-3d62797c6f35" (UID: "959366ef-3a07-43f5-bf5b-3d62797c6f35"). InnerVolumeSpecName "kube-api-access-w7zlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:50:09.270111 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.270073 2564 generic.go:356] "Generic (PLEG): container finished" podID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerID="ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602" exitCode=137 Jun 11 20:50:09.270111 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.270110 2564 generic.go:356] "Generic (PLEG): container finished" podID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerID="89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4" exitCode=137 Jun 11 20:50:09.270330 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.270163 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" Jun 11 20:50:09.270330 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.270157 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerDied","Data":"ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602"} Jun 11 20:50:09.270330 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.270237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerDied","Data":"89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4"} Jun 11 20:50:09.270330 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.270260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s" event={"ID":"959366ef-3a07-43f5-bf5b-3d62797c6f35","Type":"ContainerDied","Data":"4044fc3c904885c438297c022b50f5415353036c48cf68128189ab194a401dae"} Jun 11 20:50:09.270330 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.270290 2564 scope.go:117] "RemoveContainer" containerID="0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe" Jun 11 20:50:09.279083 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.279065 2564 scope.go:117] "RemoveContainer" containerID="ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602" Jun 11 20:50:09.286527 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.286509 2564 scope.go:117] "RemoveContainer" containerID="8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3" Jun 11 20:50:09.292079 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.292055 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s"] Jun 11 20:50:09.295060 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.295039 2564 scope.go:117] "RemoveContainer" containerID="89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4" Jun 11 20:50:09.298243 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.298219 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-rsx5s"] Jun 11 20:50:09.303256 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.303236 2564 scope.go:117] "RemoveContainer" containerID="cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7" Jun 11 20:50:09.310675 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.310655 2564 scope.go:117] "RemoveContainer" containerID="70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6" Jun 11 20:50:09.317878 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.317863 2564 scope.go:117] "RemoveContainer" containerID="0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe" Jun 11 20:50:09.318138 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:50:09.318121 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe\": container with ID starting with 0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe not found: ID does not exist" containerID="0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe" Jun 11 20:50:09.318196 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.318147 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe"} err="failed to get container status \"0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe\": rpc error: code = NotFound desc = could not find container \"0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe\": container with ID starting with 0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe not found: ID does not exist" Jun 11 20:50:09.318196 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.318167 2564 scope.go:117] "RemoveContainer" containerID="ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602" Jun 11 20:50:09.318388 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:50:09.318372 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602\": container with ID starting with ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602 not found: ID does not exist" containerID="ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602" Jun 11 20:50:09.318426 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.318397 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602"} err="failed to get container status \"ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602\": rpc error: code = NotFound desc = could not find container \"ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602\": container with ID starting with ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602 not found: ID does not exist" Jun 11 20:50:09.318426 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.318417 2564 scope.go:117] "RemoveContainer" containerID="8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3" Jun 11 20:50:09.318650 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:50:09.318630 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3\": container with ID starting with 8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3 not found: ID does not exist" containerID="8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3" Jun 11 20:50:09.318708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.318654 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3"} err="failed to get container status \"8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3\": rpc error: code = NotFound desc = could not find container \"8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3\": container with ID starting with 8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3 not found: ID does not exist" Jun 11 20:50:09.318708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.318668 2564 scope.go:117] "RemoveContainer" containerID="89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4" Jun 11 20:50:09.318898 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:50:09.318877 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4\": container with ID starting with 89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4 not found: ID does not exist" containerID="89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4" Jun 11 20:50:09.318940 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.318903 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4"} err="failed to get container status \"89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4\": rpc error: code = NotFound desc = could not find container \"89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4\": container with ID starting with 89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4 not found: ID does not exist" Jun 11 20:50:09.318940 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.318916 2564 scope.go:117] "RemoveContainer" containerID="cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7" Jun 11 20:50:09.319172 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:50:09.319153 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7\": container with ID starting with cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7 not found: ID does not exist" containerID="cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7" Jun 11 20:50:09.319222 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.319176 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7"} err="failed to get container status \"cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7\": rpc error: code = NotFound desc = could not find container \"cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7\": container with ID starting with cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7 not found: ID does not exist" Jun 11 20:50:09.319222 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.319191 2564 scope.go:117] "RemoveContainer" containerID="70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6" Jun 11 20:50:09.319439 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:50:09.319418 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6\": container with ID starting with 70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6 not found: ID does not exist" containerID="70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6" Jun 11 20:50:09.319486 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.319443 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6"} err="failed to get container status \"70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6\": rpc error: code = NotFound desc = could not find container \"70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6\": container with ID starting with 70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6 not found: ID does not exist" Jun 11 20:50:09.319486 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.319461 2564 scope.go:117] "RemoveContainer" containerID="0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe" Jun 11 20:50:09.319689 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.319673 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe"} err="failed to get container status \"0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe\": rpc error: code = NotFound desc = could not find container \"0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe\": container with ID starting with 0deac89ff5881b728cd6ec83fc2b337de699aab6be400563de6a6f267e742bbe not found: ID does not exist" Jun 11 20:50:09.319739 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.319690 2564 scope.go:117] "RemoveContainer" containerID="ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602" Jun 11 20:50:09.319936 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.319919 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602"} err="failed to get container status \"ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602\": rpc error: code = NotFound desc = could not find container \"ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602\": container with ID starting with ba47266369af26d7308617ec55b94c3235fd45d4098177f729e2062dd4346602 not found: ID does not exist" Jun 11 20:50:09.320048 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.319943 2564 scope.go:117] "RemoveContainer" containerID="8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3" Jun 11 20:50:09.320188 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.320170 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3"} err="failed to get container status \"8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3\": rpc error: code = NotFound desc = could not find container \"8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3\": container with ID starting with 8aeb2c3fdd2d1af995f3c59da607d6067505dd78ffa6360a4cdc61656aab11a3 not found: ID does not exist" Jun 11 20:50:09.320237 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.320188 2564 scope.go:117] "RemoveContainer" containerID="89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4" Jun 11 20:50:09.320386 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.320368 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4"} err="failed to get container status \"89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4\": rpc error: code = NotFound desc = could not find container \"89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4\": container with ID starting with 89a3ec10147178b16f9f23339ad8549823bee77a2ef68edf13192e1d456e0fc4 not found: ID does not exist" Jun 11 20:50:09.320426 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.320386 2564 scope.go:117] "RemoveContainer" containerID="cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7" Jun 11 20:50:09.320577 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.320561 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7"} err="failed to get container status \"cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7\": rpc error: code = NotFound desc = could not find container \"cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7\": container with ID starting with cb6234d29e8c812234f4553f3ce60144ad3886a3a2dc029a5479cf6f64dcedd7 not found: ID does not exist" Jun 11 20:50:09.320637 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.320576 2564 scope.go:117] "RemoveContainer" containerID="70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6" Jun 11 20:50:09.320789 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.320774 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6"} err="failed to get container status \"70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6\": rpc error: code = NotFound desc = could not find container \"70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6\": container with ID starting with 70d2600974bde35874754ca2b20412fcbe00b05a35e24021dbdab5a40c8e8cc6 not found: ID does not exist" Jun 11 20:50:09.362283 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.362247 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/959366ef-3a07-43f5-bf5b-3d62797c6f35-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:50:09.362283 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.362278 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7zlf\" (UniqueName: \"kubernetes.io/projected/959366ef-3a07-43f5-bf5b-3d62797c6f35-kube-api-access-w7zlf\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:50:09.362283 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.362287 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:50:09.362482 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.362295 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:50:09.362482 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.362305 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-ui-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:50:09.362482 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:09.362312 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/959366ef-3a07-43f5-bf5b-3d62797c6f35-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:50:11.075645 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:11.075605 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" path="/var/lib/kubelet/pods/959366ef-3a07-43f5-bf5b-3d62797c6f35/volumes" Jun 11 20:50:35.137155 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:35.137127 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:50:35.139478 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:35.139454 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:50:35.143688 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:35.143665 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:50:35.145474 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:50:35.145457 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:51:02.484852 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.484778 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878"] Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485115 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="online" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485128 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="online" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485139 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="registry" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485147 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="registry" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485157 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="feast-init" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485162 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="feast-init" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485167 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="feast-apply" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485172 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="feast-apply" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485180 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="offline" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485185 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="offline" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485196 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="ui" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485201 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="ui" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485248 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="online" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485255 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="registry" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485261 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="offline" Jun 11 20:51:02.487447 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.485267 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="959366ef-3a07-43f5-bf5b-3d62797c6f35" containerName="ui" Jun 11 20:51:02.488407 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.488391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.490587 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.490555 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 11 20:51:02.490803 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.490780 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 11 20:51:02.490924 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.490813 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-g7pnv\"" Jun 11 20:51:02.490924 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.490558 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 11 20:51:02.490924 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.490734 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 11 20:51:02.491115 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.490558 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 11 20:51:02.491233 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.491216 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 11 20:51:02.499408 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.499384 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878"] Jun 11 20:51:02.605091 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.605057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.605288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.605105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13f86281-b043-4faa-aae4-432bce428497-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.605288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.605135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.605288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.605164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfc8\" (UniqueName: \"kubernetes.io/projected/13f86281-b043-4faa-aae4-432bce428497-kube-api-access-hxfc8\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.605288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.605202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.605288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.605228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13f86281-b043-4faa-aae4-432bce428497-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.705816 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.705779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfc8\" (UniqueName: \"kubernetes.io/projected/13f86281-b043-4faa-aae4-432bce428497-kube-api-access-hxfc8\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.706081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.705832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.706081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.705860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13f86281-b043-4faa-aae4-432bce428497-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.706081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.705926 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.706081 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:51:02.705992 2564 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jun 11 20:51:02.706351 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.705994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13f86281-b043-4faa-aae4-432bce428497-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.706351 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:51:02.706094 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-online-tls podName:13f86281-b043-4faa-aae4-432bce428497 nodeName:}" failed. No retries permitted until 2026-06-11 20:51:03.20607179 +0000 UTC m=+1528.560818095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-online-tls") pod "feast-simple-feast-remote-setup-669bcf45fc-f9878" (UID: "13f86281-b043-4faa-aae4-432bce428497") : secret "feast-simple-feast-remote-setup-online-tls" not found Jun 11 20:51:02.706351 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.706141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.706351 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:51:02.706242 2564 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 11 20:51:02.706351 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:51:02.706276 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-offline-tls podName:13f86281-b043-4faa-aae4-432bce428497 nodeName:}" failed. No retries permitted until 2026-06-11 20:51:03.206264007 +0000 UTC m=+1528.561010306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-offline-tls") pod "feast-simple-feast-remote-setup-669bcf45fc-f9878" (UID: "13f86281-b043-4faa-aae4-432bce428497") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 11 20:51:02.706618 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.706436 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13f86281-b043-4faa-aae4-432bce428497-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.706676 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.706633 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13f86281-b043-4faa-aae4-432bce428497-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.708773 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.708746 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:02.714455 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:02.714429 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfc8\" (UniqueName: \"kubernetes.io/projected/13f86281-b043-4faa-aae4-432bce428497-kube-api-access-hxfc8\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:03.211990 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:03.211933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:03.212186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:03.212015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:03.214384 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:03.214355 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:03.214384 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:03.214375 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-f9878\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:03.400144 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:03.400101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:03.539231 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:03.539205 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878"] Jun 11 20:51:03.541440 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:51:03.541400 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13f86281_b043_4faa_aae4_432bce428497.slice/crio-cda85bf76a7714b075d26e21b8405d37c1ea2afe7e6cad538a880652e9c4455f WatchSource:0}: Error finding container cda85bf76a7714b075d26e21b8405d37c1ea2afe7e6cad538a880652e9c4455f: Status 404 returned error can't find the container with id cda85bf76a7714b075d26e21b8405d37c1ea2afe7e6cad538a880652e9c4455f Jun 11 20:51:04.493337 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:04.493297 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerStarted","Data":"a736c842e97a465d64b2b39dbc56f466805387bcc23a801ffc7a772b845edf73"} Jun 11 20:51:04.493337 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:04.493345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerStarted","Data":"cda85bf76a7714b075d26e21b8405d37c1ea2afe7e6cad538a880652e9c4455f"} Jun 11 20:51:08.511142 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:08.511105 2564 generic.go:356] "Generic (PLEG): container finished" podID="13f86281-b043-4faa-aae4-432bce428497" containerID="a736c842e97a465d64b2b39dbc56f466805387bcc23a801ffc7a772b845edf73" exitCode=0 Jun 11 20:51:08.511538 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:08.511188 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerDied","Data":"a736c842e97a465d64b2b39dbc56f466805387bcc23a801ffc7a772b845edf73"} Jun 11 20:51:09.517458 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:09.517421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerStarted","Data":"a6f296ccd99b2fc6346c00f228541c8da4fed3a2111f667a9ac7dbf0d7bc9f96"} Jun 11 20:51:14.541040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:14.541007 2564 generic.go:356] "Generic (PLEG): container finished" podID="13f86281-b043-4faa-aae4-432bce428497" containerID="a6f296ccd99b2fc6346c00f228541c8da4fed3a2111f667a9ac7dbf0d7bc9f96" exitCode=0 Jun 11 20:51:14.541404 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:14.541084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerDied","Data":"a6f296ccd99b2fc6346c00f228541c8da4fed3a2111f667a9ac7dbf0d7bc9f96"} Jun 11 20:51:15.549750 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:15.549709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerStarted","Data":"015436fcfb73513330b1c362a5639143a2dd0592037befb380eed9a10831a6dc"} Jun 11 20:51:15.550254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:15.549771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerStarted","Data":"aec6a1ca95652339a9e283c5a4f171e602338af8eb3a8aaf0f6cea24e7ed8d44"} Jun 11 20:51:15.550254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:15.549790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerStarted","Data":"3054913e5c5525504a11f48de2bcdf5b5a4d628c08417255dd17c51dad2a5835"} Jun 11 20:51:15.572080 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:15.572021 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podStartSLOduration=13.572002046 podStartE2EDuration="13.572002046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:51:15.569625424 +0000 UTC m=+1540.924371754" watchObservedRunningTime="2026-06-11 20:51:15.572002046 +0000 UTC m=+1540.926748361" Jun 11 20:51:18.400399 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:18.400351 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:18.400780 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:18.400426 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:18.400780 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:18.400447 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:18.402500 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:18.402469 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Jun 11 20:51:18.402602 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:18.402500 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.37:8816: connect: connection refused" Jun 11 20:51:18.402602 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:18.402529 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jun 11 20:51:21.401486 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:21.401450 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:21.402000 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:21.401909 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:21.402000 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:21.401949 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:21.402239 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:21.402219 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:21.405850 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:21.405824 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:21.578840 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:21.578802 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:21.579074 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:21.578852 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:21.579325 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:21.579307 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:51:21.581743 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:51:21.581724 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:52:23.920031 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:23.919929 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878"] Jun 11 20:52:23.920571 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:23.920223 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="online" containerID="cri-o://3054913e5c5525504a11f48de2bcdf5b5a4d628c08417255dd17c51dad2a5835" gracePeriod=30 Jun 11 20:52:23.920571 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:23.920287 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="ui" containerID="cri-o://015436fcfb73513330b1c362a5639143a2dd0592037befb380eed9a10831a6dc" gracePeriod=30 Jun 11 20:52:23.920571 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:23.920393 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="offline" containerID="cri-o://aec6a1ca95652339a9e283c5a4f171e602338af8eb3a8aaf0f6cea24e7ed8d44" gracePeriod=30 Jun 11 20:52:24.070747 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.070711 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc"] Jun 11 20:52:24.070952 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.070755 2564 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" secret="" err="secret \"feast-simple-feast-setup-dockercfg-z8vg4\" not found" Jun 11 20:52:24.131753 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.131716 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:24.131945 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.131782 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:24.631766957 +0000 UTC m=+1609.986513251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:24.131945 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.131783 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:24.131945 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.131724 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:24.131945 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.131723 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:24.131945 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.131845 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:24.631826519 +0000 UTC m=+1609.986572825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:24.131945 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.131869 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:24.631851432 +0000 UTC m=+1609.986597725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:24.131945 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.131888 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:24.631879917 +0000 UTC m=+1609.986626217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:24.636235 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.636208 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:24.636365 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.636284 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:25.636266456 +0000 UTC m=+1610.991012750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:24.636365 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.636211 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:24.636365 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.636353 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:25.636337743 +0000 UTC m=+1610.991084041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:24.636365 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.636211 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:24.636580 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.636387 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:25.636380504 +0000 UTC m=+1610.991126800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:24.636580 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.636211 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:24.636580 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:24.636411 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:25.636406718 +0000 UTC m=+1610.991153012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:24.844320 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.844285 2564 generic.go:356] "Generic (PLEG): container finished" podID="13f86281-b043-4faa-aae4-432bce428497" containerID="015436fcfb73513330b1c362a5639143a2dd0592037befb380eed9a10831a6dc" exitCode=0 Jun 11 20:52:24.844320 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.844312 2564 generic.go:356] "Generic (PLEG): container finished" podID="13f86281-b043-4faa-aae4-432bce428497" containerID="3054913e5c5525504a11f48de2bcdf5b5a4d628c08417255dd17c51dad2a5835" exitCode=0 Jun 11 20:52:24.844610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.844360 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerDied","Data":"015436fcfb73513330b1c362a5639143a2dd0592037befb380eed9a10831a6dc"} Jun 11 20:52:24.844610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.844399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerDied","Data":"3054913e5c5525504a11f48de2bcdf5b5a4d628c08417255dd17c51dad2a5835"} Jun 11 20:52:24.844610 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.844585 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="registry" containerID="cri-o://d3a1e3ad907bfcf8b67d9af1825df7a4f78bc283d2024e00b63860b537f89deb" gracePeriod=30 Jun 11 20:52:24.844747 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.844629 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="offline" containerID="cri-o://3d97104e5a5f472174c6a7d7b0bf684620088ce51df42e3bfc102acad63b7eeb" gracePeriod=30 Jun 11 20:52:24.844747 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.844630 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="online" containerID="cri-o://50c206469b661fe7f2ae6cbb6f58dd975e848df75014088ea091689cc8936b56" gracePeriod=30 Jun 11 20:52:24.844847 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:24.844630 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="ui" containerID="cri-o://654bfcbedcfff96baf932bc3014b1e2261ab7639d0f7e49e829431067ff98df6" gracePeriod=30 Jun 11 20:52:25.647106 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:25.647017 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:25.647106 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:25.647092 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:27.647076845 +0000 UTC m=+1613.001823139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:25.647570 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:25.647020 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:25.647570 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:25.647178 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:27.647157925 +0000 UTC m=+1613.001904235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:25.647570 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:25.647017 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:25.647570 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:25.647254 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:27.647237409 +0000 UTC m=+1613.001983706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:25.647570 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:25.647021 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:25.647570 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:25.647302 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:27.64729112 +0000 UTC m=+1613.002037420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:25.851284 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:25.851255 2564 generic.go:356] "Generic (PLEG): container finished" podID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerID="654bfcbedcfff96baf932bc3014b1e2261ab7639d0f7e49e829431067ff98df6" exitCode=0 Jun 11 20:52:25.851284 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:25.851276 2564 generic.go:356] "Generic (PLEG): container finished" podID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerID="50c206469b661fe7f2ae6cbb6f58dd975e848df75014088ea091689cc8936b56" exitCode=0 Jun 11 20:52:25.851491 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:25.851304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerDied","Data":"654bfcbedcfff96baf932bc3014b1e2261ab7639d0f7e49e829431067ff98df6"} Jun 11 20:52:25.851491 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:25.851333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerDied","Data":"50c206469b661fe7f2ae6cbb6f58dd975e848df75014088ea091689cc8936b56"} Jun 11 20:52:27.668761 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:27.668718 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:27.669207 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:27.668802 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:31.668786731 +0000 UTC m=+1617.023533030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:27.669207 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:27.668718 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:27.669207 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:27.668882 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:31.668868081 +0000 UTC m=+1617.023614380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:27.669207 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:27.668734 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:27.669207 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:27.668737 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:27.669207 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:27.668921 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:31.668909595 +0000 UTC m=+1617.023655895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:27.669207 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:27.668997 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:31.668958681 +0000 UTC m=+1617.023704975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:28.548256 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:28.548201 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" Jun 11 20:52:29.227333 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:29.227290 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jun 11 20:52:31.402752 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:31.402710 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Jun 11 20:52:31.579271 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:31.579228 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jun 11 20:52:31.707002 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:31.706887 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:31.707002 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:31.706982 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:39.706949453 +0000 UTC m=+1625.061695750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:31.707002 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:31.706897 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:31.707257 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:31.707024 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:39.707014402 +0000 UTC m=+1625.061760695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:31.707257 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:31.706897 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:31.707257 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:31.707057 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:39.707050091 +0000 UTC m=+1625.061796385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:31.707257 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:31.706989 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:31.707257 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:31.707093 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:39.707085781 +0000 UTC m=+1625.061832075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:38.547629 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:38.547579 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" Jun 11 20:52:39.228050 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:39.228006 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jun 11 20:52:39.787739 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:39.787704 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:39.788148 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:39.787749 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:39.788148 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:39.787775 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:55.787760101 +0000 UTC m=+1641.142506394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-registry-tls" not found Jun 11 20:52:39.788148 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:39.787705 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:39.788148 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:39.787704 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:39.788148 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:39.787815 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:55.787797724 +0000 UTC m=+1641.142544018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:52:39.788148 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:39.787902 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:55.787894159 +0000 UTC m=+1641.142640453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:52:39.788148 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:52:39.787912 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls podName:025cce3d-b61e-41a1-b58e-1c7fcbb636b6 nodeName:}" failed. No retries permitted until 2026-06-11 20:52:55.78790695 +0000 UTC m=+1641.142653244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls") pod "feast-simple-feast-setup-75c64c888b-mw7gc" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6") : secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:52:41.403021 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:41.402947 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Jun 11 20:52:41.579054 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:41.579004 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jun 11 20:52:48.547765 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:48.547716 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" Jun 11 20:52:48.548250 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:48.547879 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:52:49.227526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:49.227480 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jun 11 20:52:49.227697 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:49.227618 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:52:51.402546 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:51.402496 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Jun 11 20:52:51.403042 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:51.402645 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:52:51.579125 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:51.579083 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jun 11 20:52:51.579325 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:51.579203 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:52:53.977663 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:53.977616 2564 generic.go:356] "Generic (PLEG): container finished" podID="13f86281-b043-4faa-aae4-432bce428497" containerID="aec6a1ca95652339a9e283c5a4f171e602338af8eb3a8aaf0f6cea24e7ed8d44" exitCode=137 Jun 11 20:52:53.977663 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:53.977665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerDied","Data":"aec6a1ca95652339a9e283c5a4f171e602338af8eb3a8aaf0f6cea24e7ed8d44"} Jun 11 20:52:54.582613 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.582581 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:52:54.622200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.622166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-offline-tls\") pod \"13f86281-b043-4faa-aae4-432bce428497\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " Jun 11 20:52:54.622200 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.622203 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxfc8\" (UniqueName: \"kubernetes.io/projected/13f86281-b043-4faa-aae4-432bce428497-kube-api-access-hxfc8\") pod \"13f86281-b043-4faa-aae4-432bce428497\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " Jun 11 20:52:54.622469 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.622219 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13f86281-b043-4faa-aae4-432bce428497-registry-tls\") pod \"13f86281-b043-4faa-aae4-432bce428497\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " Jun 11 20:52:54.622469 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.622289 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-online-tls\") pod \"13f86281-b043-4faa-aae4-432bce428497\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " Jun 11 20:52:54.622469 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.622323 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-ui-tls\") pod \"13f86281-b043-4faa-aae4-432bce428497\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " Jun 11 20:52:54.622469 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.622374 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13f86281-b043-4faa-aae4-432bce428497-feast-data\") pod \"13f86281-b043-4faa-aae4-432bce428497\" (UID: \"13f86281-b043-4faa-aae4-432bce428497\") " Jun 11 20:52:54.622711 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.622681 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f86281-b043-4faa-aae4-432bce428497-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "13f86281-b043-4faa-aae4-432bce428497" (UID: "13f86281-b043-4faa-aae4-432bce428497"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:52:54.623096 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.623070 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f86281-b043-4faa-aae4-432bce428497-feast-data" (OuterVolumeSpecName: "feast-data") pod "13f86281-b043-4faa-aae4-432bce428497" (UID: "13f86281-b043-4faa-aae4-432bce428497"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:52:54.625218 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.625185 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "13f86281-b043-4faa-aae4-432bce428497" (UID: "13f86281-b043-4faa-aae4-432bce428497"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:52:54.625312 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.625238 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-online-tls" (OuterVolumeSpecName: "online-tls") pod "13f86281-b043-4faa-aae4-432bce428497" (UID: "13f86281-b043-4faa-aae4-432bce428497"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:52:54.625312 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.625251 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "13f86281-b043-4faa-aae4-432bce428497" (UID: "13f86281-b043-4faa-aae4-432bce428497"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:52:54.625312 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.625250 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f86281-b043-4faa-aae4-432bce428497-kube-api-access-hxfc8" (OuterVolumeSpecName: "kube-api-access-hxfc8") pod "13f86281-b043-4faa-aae4-432bce428497" (UID: "13f86281-b043-4faa-aae4-432bce428497"). InnerVolumeSpecName "kube-api-access-hxfc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:52:54.723665 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.723575 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:54.723665 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.723601 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-ui-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:54.723665 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.723610 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13f86281-b043-4faa-aae4-432bce428497-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:54.723665 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.723620 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/13f86281-b043-4faa-aae4-432bce428497-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:54.723665 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.723628 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxfc8\" (UniqueName: \"kubernetes.io/projected/13f86281-b043-4faa-aae4-432bce428497-kube-api-access-hxfc8\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:54.723665 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.723636 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/13f86281-b043-4faa-aae4-432bce428497-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:54.984660 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.984576 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" Jun 11 20:52:54.985151 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.984581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878" event={"ID":"13f86281-b043-4faa-aae4-432bce428497","Type":"ContainerDied","Data":"cda85bf76a7714b075d26e21b8405d37c1ea2afe7e6cad538a880652e9c4455f"} Jun 11 20:52:54.985151 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.984727 2564 scope.go:117] "RemoveContainer" containerID="015436fcfb73513330b1c362a5639143a2dd0592037befb380eed9a10831a6dc" Jun 11 20:52:54.988480 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.988449 2564 generic.go:356] "Generic (PLEG): container finished" podID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerID="3d97104e5a5f472174c6a7d7b0bf684620088ce51df42e3bfc102acad63b7eeb" exitCode=137 Jun 11 20:52:54.988480 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.988481 2564 generic.go:356] "Generic (PLEG): container finished" podID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerID="d3a1e3ad907bfcf8b67d9af1825df7a4f78bc283d2024e00b63860b537f89deb" exitCode=137 Jun 11 20:52:54.988682 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.988483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerDied","Data":"3d97104e5a5f472174c6a7d7b0bf684620088ce51df42e3bfc102acad63b7eeb"} Jun 11 20:52:54.988682 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.988527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerDied","Data":"d3a1e3ad907bfcf8b67d9af1825df7a4f78bc283d2024e00b63860b537f89deb"} Jun 11 20:52:54.994441 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:54.994423 2564 scope.go:117] "RemoveContainer" containerID="aec6a1ca95652339a9e283c5a4f171e602338af8eb3a8aaf0f6cea24e7ed8d44" Jun 11 20:52:55.002481 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.002462 2564 scope.go:117] "RemoveContainer" containerID="3054913e5c5525504a11f48de2bcdf5b5a4d628c08417255dd17c51dad2a5835" Jun 11 20:52:55.007016 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.006990 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878"] Jun 11 20:52:55.010438 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.010415 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-f9878"] Jun 11 20:52:55.010762 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.010746 2564 scope.go:117] "RemoveContainer" containerID="a6f296ccd99b2fc6346c00f228541c8da4fed3a2111f667a9ac7dbf0d7bc9f96" Jun 11 20:52:55.018140 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.018119 2564 scope.go:117] "RemoveContainer" containerID="a736c842e97a465d64b2b39dbc56f466805387bcc23a801ffc7a772b845edf73" Jun 11 20:52:55.075542 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.075504 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f86281-b043-4faa-aae4-432bce428497" path="/var/lib/kubelet/pods/13f86281-b043-4faa-aae4-432bce428497/volumes" Jun 11 20:52:55.505527 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.505503 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:52:55.633354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.633263 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79rvx\" (UniqueName: \"kubernetes.io/projected/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-kube-api-access-79rvx\") pod \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " Jun 11 20:52:55.633354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.633311 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls\") pod \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " Jun 11 20:52:55.633354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.633342 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls\") pod \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " Jun 11 20:52:55.633628 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.633476 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls\") pod \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " Jun 11 20:52:55.633628 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.633516 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls\") pod \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " Jun 11 20:52:55.633628 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.633622 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-feast-data\") pod \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\" (UID: \"025cce3d-b61e-41a1-b58e-1c7fcbb636b6\") " Jun 11 20:52:55.634228 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.634196 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-feast-data" (OuterVolumeSpecName: "feast-data") pod "025cce3d-b61e-41a1-b58e-1c7fcbb636b6" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:52:55.635629 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.635606 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-kube-api-access-79rvx" (OuterVolumeSpecName: "kube-api-access-79rvx") pod "025cce3d-b61e-41a1-b58e-1c7fcbb636b6" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6"). InnerVolumeSpecName "kube-api-access-79rvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:52:55.635937 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.635913 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "025cce3d-b61e-41a1-b58e-1c7fcbb636b6" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:52:55.636016 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.635931 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "025cce3d-b61e-41a1-b58e-1c7fcbb636b6" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:52:55.636065 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.636011 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "025cce3d-b61e-41a1-b58e-1c7fcbb636b6" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:52:55.636102 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.636082 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls" (OuterVolumeSpecName: "online-tls") pod "025cce3d-b61e-41a1-b58e-1c7fcbb636b6" (UID: "025cce3d-b61e-41a1-b58e-1c7fcbb636b6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:52:55.734245 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.734204 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:55.734245 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.734237 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:55.734245 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.734248 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:55.734484 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.734259 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:55.734484 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.734271 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rvx\" (UniqueName: \"kubernetes.io/projected/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-kube-api-access-79rvx\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:55.734484 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.734282 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/025cce3d-b61e-41a1-b58e-1c7fcbb636b6-ui-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:52:55.995452 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.995362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" event={"ID":"025cce3d-b61e-41a1-b58e-1c7fcbb636b6","Type":"ContainerDied","Data":"c7aafaf50ed8450bf354d53750f9711e6025e5b668a7ee79954aa1fd251dfb21"} Jun 11 20:52:55.995452 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.995394 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc" Jun 11 20:52:55.995452 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:55.995429 2564 scope.go:117] "RemoveContainer" containerID="654bfcbedcfff96baf932bc3014b1e2261ab7639d0f7e49e829431067ff98df6" Jun 11 20:52:56.004748 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:56.004729 2564 scope.go:117] "RemoveContainer" containerID="3d97104e5a5f472174c6a7d7b0bf684620088ce51df42e3bfc102acad63b7eeb" Jun 11 20:52:56.012097 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:56.012076 2564 scope.go:117] "RemoveContainer" containerID="50c206469b661fe7f2ae6cbb6f58dd975e848df75014088ea091689cc8936b56" Jun 11 20:52:56.017325 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:56.017304 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc"] Jun 11 20:52:56.021313 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:56.021296 2564 scope.go:117] "RemoveContainer" containerID="d3a1e3ad907bfcf8b67d9af1825df7a4f78bc283d2024e00b63860b537f89deb" Jun 11 20:52:56.021592 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:56.021572 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-mw7gc"] Jun 11 20:52:56.029156 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:56.029128 2564 scope.go:117] "RemoveContainer" containerID="b977815f2e5c843df78d30cf18e886786d386d851e24472e6a6755b3290cd8e8" Jun 11 20:52:56.036726 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:56.036709 2564 scope.go:117] "RemoveContainer" containerID="4981a4c89488f27de252ed1b5d196be1af14ac1e8e870913e579cca8df039cf1" Jun 11 20:52:57.080031 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:52:57.079997 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" path="/var/lib/kubelet/pods/025cce3d-b61e-41a1-b58e-1c7fcbb636b6/volumes" Jun 11 20:53:33.456133 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456090 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns"] Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456511 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="feast-init" Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456525 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="feast-init" Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456539 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="offline" Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456547 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="offline" Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456565 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="feast-init" Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456574 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="feast-init" Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456586 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="online" Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456595 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="online" Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456609 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="online" Jun 11 20:53:33.456611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456617 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="online" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456628 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="registry" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456637 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="registry" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456648 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="feast-apply" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456656 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="feast-apply" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456667 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="feast-apply" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456675 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="feast-apply" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456684 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="ui" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456691 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="ui" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456700 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="ui" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456710 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="ui" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456724 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="offline" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456732 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="offline" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456793 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="online" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456802 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="ui" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456811 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="offline" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456821 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="registry" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456829 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f86281-b043-4faa-aae4-432bce428497" containerName="ui" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456837 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="offline" Jun 11 20:53:33.456958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.456845 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="025cce3d-b61e-41a1-b58e-1c7fcbb636b6" containerName="online" Jun 11 20:53:33.460554 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.460535 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.462869 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.462846 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 11 20:53:33.463260 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.463241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 11 20:53:33.463428 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.463415 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 11 20:53:33.463606 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.463594 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 11 20:53:33.463760 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.463748 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-ntbrz\"" Jun 11 20:53:33.463899 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.463887 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 11 20:53:33.464083 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.464070 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 11 20:53:33.471808 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.471780 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns"] Jun 11 20:53:33.552461 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.552422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.552639 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.552509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.552639 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.552537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w29xb\" (UniqueName: \"kubernetes.io/projected/b4eb1313-ac51-4bd3-824c-4408bbba348e-kube-api-access-w29xb\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.552639 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.552597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eb1313-ac51-4bd3-824c-4408bbba348e-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.552781 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.552645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.552781 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.552726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.653924 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.653884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.653924 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.653926 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.654224 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.653992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.654224 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.654023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w29xb\" (UniqueName: \"kubernetes.io/projected/b4eb1313-ac51-4bd3-824c-4408bbba348e-kube-api-access-w29xb\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.654224 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.654061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eb1313-ac51-4bd3-824c-4408bbba348e-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.654224 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.654098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.654224 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:53:33.654062 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:53:33.654477 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:53:33.654336 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-offline-tls podName:b4eb1313-ac51-4bd3-824c-4408bbba348e nodeName:}" failed. No retries permitted until 2026-06-11 20:53:34.154309324 +0000 UTC m=+1679.509055633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-offline-tls") pod "feast-simple-feast-setup-75c64c888b-vb8ns" (UID: "b4eb1313-ac51-4bd3-824c-4408bbba348e") : secret "feast-simple-feast-setup-offline-tls" not found Jun 11 20:53:33.654544 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.654508 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eb1313-ac51-4bd3-824c-4408bbba348e-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.656603 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.656574 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.656723 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.656612 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.656767 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.656719 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:33.662011 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:33.661952 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w29xb\" (UniqueName: \"kubernetes.io/projected/b4eb1313-ac51-4bd3-824c-4408bbba348e-kube-api-access-w29xb\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:34.159515 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:34.159484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:34.161889 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:34.161859 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-vb8ns\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:34.378162 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:34.378122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:34.515704 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:34.515673 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns"] Jun 11 20:53:34.517425 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:53:34.517398 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4eb1313_ac51_4bd3_824c_4408bbba348e.slice/crio-e5d8c122327ce3855ef0eb2ecce93d8440d8a38af4237f5882891017a8853cdf WatchSource:0}: Error finding container e5d8c122327ce3855ef0eb2ecce93d8440d8a38af4237f5882891017a8853cdf: Status 404 returned error can't find the container with id e5d8c122327ce3855ef0eb2ecce93d8440d8a38af4237f5882891017a8853cdf Jun 11 20:53:35.154089 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:35.154051 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerStarted","Data":"1e83f634f149d8b3f02c9ca0a14d0a113e494fe8ed5f3514586bac1e01ec1749"} Jun 11 20:53:35.154272 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:35.154097 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerStarted","Data":"e5d8c122327ce3855ef0eb2ecce93d8440d8a38af4237f5882891017a8853cdf"} Jun 11 20:53:40.176046 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:40.176001 2564 generic.go:356] "Generic (PLEG): container finished" podID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerID="1e83f634f149d8b3f02c9ca0a14d0a113e494fe8ed5f3514586bac1e01ec1749" exitCode=0 Jun 11 20:53:40.176410 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:40.176071 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerDied","Data":"1e83f634f149d8b3f02c9ca0a14d0a113e494fe8ed5f3514586bac1e01ec1749"} Jun 11 20:53:41.182232 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:41.182189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerStarted","Data":"226396f6fa7254c814b05721b1bb0dfcc4e9adedfa4dca3b93b2a54ba245a76d"} Jun 11 20:53:46.208937 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:46.208845 2564 generic.go:356] "Generic (PLEG): container finished" podID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerID="226396f6fa7254c814b05721b1bb0dfcc4e9adedfa4dca3b93b2a54ba245a76d" exitCode=0 Jun 11 20:53:46.208937 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:46.208900 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerDied","Data":"226396f6fa7254c814b05721b1bb0dfcc4e9adedfa4dca3b93b2a54ba245a76d"} Jun 11 20:53:47.219198 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:47.219148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerStarted","Data":"cc964f99588885952475381d1086d2ab68c6a2cfb9fabae8f0b5d430b3a5949b"} Jun 11 20:53:47.219698 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:47.219218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerStarted","Data":"fdf1618c17995a5abb9e8c437062b13ba8c45a0733e6e239955f52809b50da9b"} Jun 11 20:53:47.219698 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:47.219238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerStarted","Data":"57da61d763fe393805f0e90002b8ce4d383f550ea2e13cb6e0deca7ce5c82bfc"} Jun 11 20:53:47.219698 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:47.219256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerStarted","Data":"6c81fb32bfd550ba16c5e13bbe5c8e4766c4d64c3580245cd91139635324cb62"} Jun 11 20:53:47.260433 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:47.260364 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podStartSLOduration=14.260340531 podStartE2EDuration="14.260340531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:53:47.259248962 +0000 UTC m=+1692.613995278" watchObservedRunningTime="2026-06-11 20:53:47.260340531 +0000 UTC m=+1692.615086850" Jun 11 20:53:49.380264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:49.379133 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:49.380264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:49.379219 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:49.380264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:49.379239 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:49.380264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:49.379257 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:49.381264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:49.381039 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" Jun 11 20:53:49.381264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:49.381076 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.38:8816: connect: connection refused" Jun 11 20:53:49.381264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:49.381201 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Jun 11 20:53:49.381624 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:49.381588 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused" Jun 11 20:53:52.380012 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:52.379921 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:52.380456 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:52.380334 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:52.380871 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:52.380846 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:52.381041 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:52.380882 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:52.381041 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:52.380894 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:52.381041 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:52.380934 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:52.381264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:52.381244 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:52.384915 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:52.384896 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:53.250843 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:53.250813 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:53.251064 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:53.250858 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:53.251242 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:53.251223 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:53:53.253664 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:53:53.253644 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:54:54.570250 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.570214 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns"] Jun 11 20:54:54.570835 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.570468 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="registry" containerID="cri-o://6c81fb32bfd550ba16c5e13bbe5c8e4766c4d64c3580245cd91139635324cb62" gracePeriod=30 Jun 11 20:54:54.570835 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.570548 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="online" containerID="cri-o://57da61d763fe393805f0e90002b8ce4d383f550ea2e13cb6e0deca7ce5c82bfc" gracePeriod=30 Jun 11 20:54:54.570835 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.570596 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="ui" containerID="cri-o://cc964f99588885952475381d1086d2ab68c6a2cfb9fabae8f0b5d430b3a5949b" gracePeriod=30 Jun 11 20:54:54.570835 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.570547 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="offline" containerID="cri-o://fdf1618c17995a5abb9e8c437062b13ba8c45a0733e6e239955f52809b50da9b" gracePeriod=30 Jun 11 20:54:54.885656 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.885506 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd"] Jun 11 20:54:54.889911 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.889886 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:54.893631 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.893607 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-hzsvh\"" Jun 11 20:54:54.907664 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.907631 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd"] Jun 11 20:54:54.962742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.962692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:54.962985 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.962752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:54.962985 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.962781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fhp\" (UniqueName: \"kubernetes.io/projected/1b55b440-dab0-45fa-b347-056928127e9d-kube-api-access-58fhp\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:54.962985 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.962843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:54.962985 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.962884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:54.962985 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:54.962913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55b440-dab0-45fa-b347-056928127e9d-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.063501 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.063451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.063712 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.063517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.063712 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.063549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fhp\" (UniqueName: \"kubernetes.io/projected/1b55b440-dab0-45fa-b347-056928127e9d-kube-api-access-58fhp\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.063712 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.063621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.063712 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.063670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.063712 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.063705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55b440-dab0-45fa-b347-056928127e9d-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.064008 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:54:55.063669 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:54:55.064008 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:54:55.063810 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls podName:1b55b440-dab0-45fa-b347-056928127e9d nodeName:}" failed. No retries permitted until 2026-06-11 20:54:55.563785841 +0000 UTC m=+1760.918532145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls") pod "feast-simple-feast-setup-75c64c888b-b86qd" (UID: "1b55b440-dab0-45fa-b347-056928127e9d") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:54:55.064129 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:54:55.063729 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:54:55.064129 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:54:55.064078 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-online-tls podName:1b55b440-dab0-45fa-b347-056928127e9d nodeName:}" failed. No retries permitted until 2026-06-11 20:54:55.564061419 +0000 UTC m=+1760.918807714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-online-tls") pod "feast-simple-feast-setup-75c64c888b-b86qd" (UID: "1b55b440-dab0-45fa-b347-056928127e9d") : secret "feast-simple-feast-setup-online-tls" not found Jun 11 20:54:55.064240 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.064216 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55b440-dab0-45fa-b347-056928127e9d-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.067227 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.067200 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.067405 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.067359 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.075190 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.075133 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fhp\" (UniqueName: \"kubernetes.io/projected/1b55b440-dab0-45fa-b347-056928127e9d-kube-api-access-58fhp\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.495742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.495662 2564 generic.go:356] "Generic (PLEG): container finished" podID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerID="cc964f99588885952475381d1086d2ab68c6a2cfb9fabae8f0b5d430b3a5949b" exitCode=0 Jun 11 20:54:55.495742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.495689 2564 generic.go:356] "Generic (PLEG): container finished" podID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerID="57da61d763fe393805f0e90002b8ce4d383f550ea2e13cb6e0deca7ce5c82bfc" exitCode=0 Jun 11 20:54:55.495742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.495731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerDied","Data":"cc964f99588885952475381d1086d2ab68c6a2cfb9fabae8f0b5d430b3a5949b"} Jun 11 20:54:55.495947 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.495773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerDied","Data":"57da61d763fe393805f0e90002b8ce4d383f550ea2e13cb6e0deca7ce5c82bfc"} Jun 11 20:54:55.567126 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.567088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.567335 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.567158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:55.567335 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:54:55.567289 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:54:55.567463 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:54:55.567388 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls podName:1b55b440-dab0-45fa-b347-056928127e9d nodeName:}" failed. No retries permitted until 2026-06-11 20:54:56.567368467 +0000 UTC m=+1761.922114766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls") pod "feast-simple-feast-setup-75c64c888b-b86qd" (UID: "1b55b440-dab0-45fa-b347-056928127e9d") : secret "feast-simple-feast-setup-ui-tls" not found Jun 11 20:54:55.569601 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:55.569574 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:56.575108 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:56.575072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:56.577544 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:56.577518 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-b86qd\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:56.700412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:56.700371 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:54:56.838107 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:56.838078 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd"] Jun 11 20:54:56.840158 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:54:56.840131 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b55b440_dab0_45fa_b347_056928127e9d.slice/crio-f2a79e74fe421352b4dc906fb0d0b443ae28ff064b3cf027df4cb7042db1c7c1 WatchSource:0}: Error finding container f2a79e74fe421352b4dc906fb0d0b443ae28ff064b3cf027df4cb7042db1c7c1: Status 404 returned error can't find the container with id f2a79e74fe421352b4dc906fb0d0b443ae28ff064b3cf027df4cb7042db1c7c1 Jun 11 20:54:57.505366 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:57.505332 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerStarted","Data":"0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0"} Jun 11 20:54:57.505515 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:54:57.505375 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerStarted","Data":"f2a79e74fe421352b4dc906fb0d0b443ae28ff064b3cf027df4cb7042db1c7c1"} Jun 11 20:55:02.381015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:02.380924 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" Jun 11 20:55:02.527132 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:02.527095 2564 generic.go:356] "Generic (PLEG): container finished" podID="1b55b440-dab0-45fa-b347-056928127e9d" containerID="0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0" exitCode=0 Jun 11 20:55:02.527306 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:02.527170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerDied","Data":"0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0"} Jun 11 20:55:03.251910 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:03.251869 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Jun 11 20:55:03.532915 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:03.532832 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerStarted","Data":"8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768"} Jun 11 20:55:08.556791 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:08.556703 2564 generic.go:356] "Generic (PLEG): container finished" podID="1b55b440-dab0-45fa-b347-056928127e9d" containerID="8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768" exitCode=0 Jun 11 20:55:08.556791 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:08.556777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerDied","Data":"8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768"} Jun 11 20:55:09.578256 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:09.578213 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerStarted","Data":"6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd"} Jun 11 20:55:09.578751 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:09.578283 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerStarted","Data":"50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c"} Jun 11 20:55:09.578751 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:09.578305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerStarted","Data":"1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544"} Jun 11 20:55:09.578751 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:09.578324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerStarted","Data":"e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0"} Jun 11 20:55:09.602509 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:09.602444 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podStartSLOduration=15.602421384 podStartE2EDuration="15.602421384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:55:09.599774871 +0000 UTC m=+1774.954521188" watchObservedRunningTime="2026-06-11 20:55:09.602421384 +0000 UTC m=+1774.957167701" Jun 11 20:55:11.700607 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:11.700505 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:11.700607 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:11.700569 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:11.700607 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:11.700592 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:11.700607 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:11.700611 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:11.702856 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:11.702816 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.39:6571: connect: connection refused" Jun 11 20:55:11.702856 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:11.702845 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Jun 11 20:55:11.703125 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:11.702866 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Jun 11 20:55:11.703125 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:11.702924 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.39:8816: connect: connection refused" Jun 11 20:55:12.381306 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:12.381259 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" Jun 11 20:55:13.251851 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:13.251804 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Jun 11 20:55:14.702473 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.702438 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:14.702863 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.702622 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:14.702863 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.702752 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:14.703334 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.703308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:14.703453 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.703351 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:14.703453 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.703370 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:14.703594 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.703454 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:14.703594 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.703517 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:14.703793 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.703776 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:14.707305 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:14.707285 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:15.611288 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:15.611261 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:15.615622 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:15.615587 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:55:22.380859 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:22.380812 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" Jun 11 20:55:22.381322 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:22.380946 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:55:23.251059 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:23.251013 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" Jun 11 20:55:23.251230 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:23.251176 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:55:24.648276 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:24.648240 2564 generic.go:356] "Generic (PLEG): container finished" podID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerID="fdf1618c17995a5abb9e8c437062b13ba8c45a0733e6e239955f52809b50da9b" exitCode=137 Jun 11 20:55:24.648276 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:24.648268 2564 generic.go:356] "Generic (PLEG): container finished" podID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerID="6c81fb32bfd550ba16c5e13bbe5c8e4766c4d64c3580245cd91139635324cb62" exitCode=137 Jun 11 20:55:24.648686 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:24.648304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerDied","Data":"fdf1618c17995a5abb9e8c437062b13ba8c45a0733e6e239955f52809b50da9b"} Jun 11 20:55:24.648686 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:24.648338 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerDied","Data":"6c81fb32bfd550ba16c5e13bbe5c8e4766c4d64c3580245cd91139635324cb62"} Jun 11 20:55:25.225800 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.225777 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:55:25.328408 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.328370 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eb1313-ac51-4bd3-824c-4408bbba348e-feast-data\") pod \"b4eb1313-ac51-4bd3-824c-4408bbba348e\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " Jun 11 20:55:25.328408 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.328412 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-offline-tls\") pod \"b4eb1313-ac51-4bd3-824c-4408bbba348e\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " Jun 11 20:55:25.328644 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.328463 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-ui-tls\") pod \"b4eb1313-ac51-4bd3-824c-4408bbba348e\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " Jun 11 20:55:25.328644 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.328513 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-online-tls\") pod \"b4eb1313-ac51-4bd3-824c-4408bbba348e\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " Jun 11 20:55:25.328644 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.328550 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w29xb\" (UniqueName: \"kubernetes.io/projected/b4eb1313-ac51-4bd3-824c-4408bbba348e-kube-api-access-w29xb\") pod \"b4eb1313-ac51-4bd3-824c-4408bbba348e\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " Jun 11 20:55:25.328644 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.328614 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-registry-tls\") pod \"b4eb1313-ac51-4bd3-824c-4408bbba348e\" (UID: \"b4eb1313-ac51-4bd3-824c-4408bbba348e\") " Jun 11 20:55:25.329078 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.329034 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4eb1313-ac51-4bd3-824c-4408bbba348e-feast-data" (OuterVolumeSpecName: "feast-data") pod "b4eb1313-ac51-4bd3-824c-4408bbba348e" (UID: "b4eb1313-ac51-4bd3-824c-4408bbba348e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:55:25.329219 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.329192 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b4eb1313-ac51-4bd3-824c-4408bbba348e-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:55:25.330859 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.330834 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-online-tls" (OuterVolumeSpecName: "online-tls") pod "b4eb1313-ac51-4bd3-824c-4408bbba348e" (UID: "b4eb1313-ac51-4bd3-824c-4408bbba348e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:55:25.331015 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.330995 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b4eb1313-ac51-4bd3-824c-4408bbba348e" (UID: "b4eb1313-ac51-4bd3-824c-4408bbba348e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:55:25.331147 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.331127 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b4eb1313-ac51-4bd3-824c-4408bbba348e" (UID: "b4eb1313-ac51-4bd3-824c-4408bbba348e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:55:25.331197 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.331148 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b4eb1313-ac51-4bd3-824c-4408bbba348e" (UID: "b4eb1313-ac51-4bd3-824c-4408bbba348e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:55:25.331366 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.331342 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4eb1313-ac51-4bd3-824c-4408bbba348e-kube-api-access-w29xb" (OuterVolumeSpecName: "kube-api-access-w29xb") pod "b4eb1313-ac51-4bd3-824c-4408bbba348e" (UID: "b4eb1313-ac51-4bd3-824c-4408bbba348e"). InnerVolumeSpecName "kube-api-access-w29xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:55:25.430647 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.430557 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:55:25.430647 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.430585 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-ui-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:55:25.430647 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.430593 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:55:25.430647 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.430606 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w29xb\" (UniqueName: \"kubernetes.io/projected/b4eb1313-ac51-4bd3-824c-4408bbba348e-kube-api-access-w29xb\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:55:25.430647 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.430619 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b4eb1313-ac51-4bd3-824c-4408bbba348e-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:55:25.656063 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.656027 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" event={"ID":"b4eb1313-ac51-4bd3-824c-4408bbba348e","Type":"ContainerDied","Data":"e5d8c122327ce3855ef0eb2ecce93d8440d8a38af4237f5882891017a8853cdf"} Jun 11 20:55:25.656063 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.656056 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns" Jun 11 20:55:25.656714 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.656085 2564 scope.go:117] "RemoveContainer" containerID="cc964f99588885952475381d1086d2ab68c6a2cfb9fabae8f0b5d430b3a5949b" Jun 11 20:55:25.665698 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.665677 2564 scope.go:117] "RemoveContainer" containerID="fdf1618c17995a5abb9e8c437062b13ba8c45a0733e6e239955f52809b50da9b" Jun 11 20:55:25.673744 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.673728 2564 scope.go:117] "RemoveContainer" containerID="57da61d763fe393805f0e90002b8ce4d383f550ea2e13cb6e0deca7ce5c82bfc" Jun 11 20:55:25.677710 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.677679 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns"] Jun 11 20:55:25.681582 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.681523 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-vb8ns"] Jun 11 20:55:25.683006 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.682992 2564 scope.go:117] "RemoveContainer" containerID="6c81fb32bfd550ba16c5e13bbe5c8e4766c4d64c3580245cd91139635324cb62" Jun 11 20:55:25.690634 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.690616 2564 scope.go:117] "RemoveContainer" containerID="226396f6fa7254c814b05721b1bb0dfcc4e9adedfa4dca3b93b2a54ba245a76d" Jun 11 20:55:25.697817 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:25.697799 2564 scope.go:117] "RemoveContainer" containerID="1e83f634f149d8b3f02c9ca0a14d0a113e494fe8ed5f3514586bac1e01ec1749" Jun 11 20:55:27.075891 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:27.075857 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" path="/var/lib/kubelet/pods/b4eb1313-ac51-4bd3-824c-4408bbba348e/volumes" Jun 11 20:55:35.163320 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:35.163290 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:55:35.166658 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:35.166632 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:55:35.169742 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:35.169717 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:55:35.175667 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:55:35.175645 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:56:18.297500 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297457 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl"] Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297774 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="feast-init" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297785 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="feast-init" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297798 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="ui" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297803 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="ui" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297812 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="offline" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297818 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="offline" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297826 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="online" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297832 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="online" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297843 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="feast-apply" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297848 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="feast-apply" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297862 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="registry" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297867 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="registry" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297918 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="offline" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297926 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="online" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297932 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="registry" Jun 11 20:56:18.298081 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.297939 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4eb1313-ac51-4bd3-824c-4408bbba348e" containerName="ui" Jun 11 20:56:18.301429 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.301393 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.303799 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.303769 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 11 20:56:18.303799 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.303773 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 11 20:56:18.304334 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.304310 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 11 20:56:18.304451 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.304371 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 11 20:56:18.304451 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.304396 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 11 20:56:18.304451 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.304412 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-cvhll\"" Jun 11 20:56:18.304630 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.304377 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 11 20:56:18.315225 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.315193 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl"] Jun 11 20:56:18.376477 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.376440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.376477 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.376481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.376724 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.376503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.376724 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.376627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.376724 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.376665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt42b\" (UniqueName: \"kubernetes.io/projected/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-kube-api-access-rt42b\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.376724 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.376687 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.477855 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.477815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.477855 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.477856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.478164 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.477947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.478164 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.478011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt42b\" (UniqueName: \"kubernetes.io/projected/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-kube-api-access-rt42b\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.478164 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.478052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.478164 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.478087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.478164 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:56:18.478132 2564 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jun 11 20:56:18.478438 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:56:18.478205 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-online-tls podName:bfb4a526-5204-46b3-8457-fe7c73d2a3e8 nodeName:}" failed. No retries permitted until 2026-06-11 20:56:18.97818331 +0000 UTC m=+1844.332929633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-online-tls") pod "feast-simple-feast-remote-setup-669bcf45fc-s8zbl" (UID: "bfb4a526-5204-46b3-8457-fe7c73d2a3e8") : secret "feast-simple-feast-remote-setup-online-tls" not found Jun 11 20:56:18.478555 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.478532 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.478703 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.478675 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.481177 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.481154 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.481325 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.481304 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.486249 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.486227 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt42b\" (UniqueName: \"kubernetes.io/projected/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-kube-api-access-rt42b\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.981955 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.981919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:18.984359 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:18.984323 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-s8zbl\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:19.212407 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:19.212359 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:19.361840 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:19.361811 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl"] Jun 11 20:56:19.363860 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:56:19.363835 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfb4a526_5204_46b3_8457_fe7c73d2a3e8.slice/crio-78dba001fae1e410fadeb9e1daac73990a3b0bb63cdaccbd77157711b37325b9 WatchSource:0}: Error finding container 78dba001fae1e410fadeb9e1daac73990a3b0bb63cdaccbd77157711b37325b9: Status 404 returned error can't find the container with id 78dba001fae1e410fadeb9e1daac73990a3b0bb63cdaccbd77157711b37325b9 Jun 11 20:56:19.885545 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:19.885509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerStarted","Data":"72bb6418fadbb4724cf4aca851b40cae65670d8cd960477a0bcb65d409c0f732"} Jun 11 20:56:19.885708 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:19.885557 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerStarted","Data":"78dba001fae1e410fadeb9e1daac73990a3b0bb63cdaccbd77157711b37325b9"} Jun 11 20:56:24.911190 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:24.911157 2564 generic.go:356] "Generic (PLEG): container finished" podID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerID="72bb6418fadbb4724cf4aca851b40cae65670d8cd960477a0bcb65d409c0f732" exitCode=0 Jun 11 20:56:24.911584 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:24.911234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerDied","Data":"72bb6418fadbb4724cf4aca851b40cae65670d8cd960477a0bcb65d409c0f732"} Jun 11 20:56:25.916762 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:25.916726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerStarted","Data":"2b0209bb01ad18a0edcb67e79955df35e21be369d018c23d66ec9f88a9cdb1dd"} Jun 11 20:56:30.939566 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:30.939528 2564 generic.go:356] "Generic (PLEG): container finished" podID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerID="2b0209bb01ad18a0edcb67e79955df35e21be369d018c23d66ec9f88a9cdb1dd" exitCode=0 Jun 11 20:56:30.939982 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:30.939600 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerDied","Data":"2b0209bb01ad18a0edcb67e79955df35e21be369d018c23d66ec9f88a9cdb1dd"} Jun 11 20:56:31.948427 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:31.948382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerStarted","Data":"199c9667c5b0ee674a703ea9bf9bde05f40bc3eeb5dafc2b0b2a1701ed196179"} Jun 11 20:56:31.948929 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:31.948447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerStarted","Data":"f1301589a1b774477e2d9bb9a3021c75958a2e5dca5fc5d3193e1c3219240982"} Jun 11 20:56:31.948929 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:31.948467 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerStarted","Data":"f30464c9077f6cd650600e80982bc01e798943646c11bc0e1e7792c403f81c8f"} Jun 11 20:56:31.970264 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:31.970203 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podStartSLOduration=13.970185171 podStartE2EDuration="13.970185171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:56:31.968454854 +0000 UTC m=+1857.323201219" watchObservedRunningTime="2026-06-11 20:56:31.970185171 +0000 UTC m=+1857.324931487" Jun 11 20:56:34.212853 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:34.212805 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:34.213519 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:34.212876 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:34.213519 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:34.212894 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:34.214861 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:34.214826 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.40:8816: connect: connection refused" Jun 11 20:56:34.215043 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:34.214826 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Jun 11 20:56:34.215043 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:34.214829 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Jun 11 20:56:37.213327 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:37.213287 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:37.213825 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:37.213751 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:37.213825 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:37.213786 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:37.214051 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:37.214028 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:37.217899 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:37.217875 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:37.983438 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:37.983401 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:37.983650 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:37.983452 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:37.983948 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:37.983922 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:56:37.986448 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:56:37.986423 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:57:39.233017 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:39.232981 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl"] Jun 11 20:57:39.233935 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:39.233849 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="online" containerID="cri-o://f30464c9077f6cd650600e80982bc01e798943646c11bc0e1e7792c403f81c8f" gracePeriod=30 Jun 11 20:57:39.234171 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:39.234130 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="ui" containerID="cri-o://199c9667c5b0ee674a703ea9bf9bde05f40bc3eeb5dafc2b0b2a1701ed196179" gracePeriod=30 Jun 11 20:57:39.234285 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:39.234198 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="offline" containerID="cri-o://f1301589a1b774477e2d9bb9a3021c75958a2e5dca5fc5d3193e1c3219240982" gracePeriod=30 Jun 11 20:57:39.388843 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:39.388801 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd"] Jun 11 20:57:39.389476 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:39.389156 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="registry" containerID="cri-o://e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0" gracePeriod=30 Jun 11 20:57:39.389476 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:39.389256 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="offline" containerID="cri-o://50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c" gracePeriod=30 Jun 11 20:57:39.389476 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:39.389351 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="online" containerID="cri-o://1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544" gracePeriod=30 Jun 11 20:57:39.389476 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:39.389358 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="ui" containerID="cri-o://6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd" gracePeriod=30 Jun 11 20:57:40.242245 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:40.242212 2564 generic.go:356] "Generic (PLEG): container finished" podID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerID="199c9667c5b0ee674a703ea9bf9bde05f40bc3eeb5dafc2b0b2a1701ed196179" exitCode=0 Jun 11 20:57:40.242245 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:40.242240 2564 generic.go:356] "Generic (PLEG): container finished" podID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerID="f30464c9077f6cd650600e80982bc01e798943646c11bc0e1e7792c403f81c8f" exitCode=0 Jun 11 20:57:40.242682 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:40.242291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerDied","Data":"199c9667c5b0ee674a703ea9bf9bde05f40bc3eeb5dafc2b0b2a1701ed196179"} Jun 11 20:57:40.242682 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:40.242345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerDied","Data":"f30464c9077f6cd650600e80982bc01e798943646c11bc0e1e7792c403f81c8f"} Jun 11 20:57:40.245164 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:40.245140 2564 generic.go:356] "Generic (PLEG): container finished" podID="1b55b440-dab0-45fa-b347-056928127e9d" containerID="6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd" exitCode=0 Jun 11 20:57:40.245164 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:40.245163 2564 generic.go:356] "Generic (PLEG): container finished" podID="1b55b440-dab0-45fa-b347-056928127e9d" containerID="1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544" exitCode=0 Jun 11 20:57:40.245282 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:40.245210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerDied","Data":"6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd"} Jun 11 20:57:40.245282 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:40.245252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerDied","Data":"1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544"} Jun 11 20:57:44.703615 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:44.703572 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Jun 11 20:57:45.612446 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:45.612401 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Jun 11 20:57:47.214487 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:47.214444 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Jun 11 20:57:47.984490 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:47.984445 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Jun 11 20:57:54.704199 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:54.704141 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Jun 11 20:57:55.611421 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:55.611380 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Jun 11 20:57:57.214705 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:57.214649 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Jun 11 20:57:57.983827 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:57:57.983778 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Jun 11 20:58:04.703705 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:04.703656 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Jun 11 20:58:04.704201 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:04.703823 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:58:05.612219 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:05.612176 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Jun 11 20:58:05.612417 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:05.612304 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:58:07.214243 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:07.214196 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Jun 11 20:58:07.214741 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:07.214322 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:58:07.984147 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:07.984103 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Jun 11 20:58:07.984358 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:07.984248 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:58:09.376043 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:09.376006 2564 generic.go:356] "Generic (PLEG): container finished" podID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerID="f1301589a1b774477e2d9bb9a3021c75958a2e5dca5fc5d3193e1c3219240982" exitCode=137 Jun 11 20:58:09.376441 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:09.376052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerDied","Data":"f1301589a1b774477e2d9bb9a3021c75958a2e5dca5fc5d3193e1c3219240982"} Jun 11 20:58:09.889540 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:09.889511 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:58:10.029490 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.029443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-registry-tls\") pod \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " Jun 11 20:58:10.029656 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.029503 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt42b\" (UniqueName: \"kubernetes.io/projected/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-kube-api-access-rt42b\") pod \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " Jun 11 20:58:10.029656 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.029535 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-feast-data\") pod \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " Jun 11 20:58:10.029656 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.029570 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-online-tls\") pod \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " Jun 11 20:58:10.029656 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.029648 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-offline-tls\") pod \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " Jun 11 20:58:10.029890 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.029694 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-ui-tls\") pod \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\" (UID: \"bfb4a526-5204-46b3-8457-fe7c73d2a3e8\") " Jun 11 20:58:10.029890 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.029853 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bfb4a526-5204-46b3-8457-fe7c73d2a3e8" (UID: "bfb4a526-5204-46b3-8457-fe7c73d2a3e8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 11 20:58:10.030017 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.029943 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.030083 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.030032 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-feast-data" (OuterVolumeSpecName: "feast-data") pod "bfb4a526-5204-46b3-8457-fe7c73d2a3e8" (UID: "bfb4a526-5204-46b3-8457-fe7c73d2a3e8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:58:10.032273 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.032233 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "bfb4a526-5204-46b3-8457-fe7c73d2a3e8" (UID: "bfb4a526-5204-46b3-8457-fe7c73d2a3e8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:58:10.032433 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.032324 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-online-tls" (OuterVolumeSpecName: "online-tls") pod "bfb4a526-5204-46b3-8457-fe7c73d2a3e8" (UID: "bfb4a526-5204-46b3-8457-fe7c73d2a3e8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:58:10.032433 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.032348 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bfb4a526-5204-46b3-8457-fe7c73d2a3e8" (UID: "bfb4a526-5204-46b3-8457-fe7c73d2a3e8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:58:10.032433 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.032350 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-kube-api-access-rt42b" (OuterVolumeSpecName: "kube-api-access-rt42b") pod "bfb4a526-5204-46b3-8457-fe7c73d2a3e8" (UID: "bfb4a526-5204-46b3-8457-fe7c73d2a3e8"). InnerVolumeSpecName "kube-api-access-rt42b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:58:10.036651 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.036631 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:58:10.130910 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.130866 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls\") pod \"1b55b440-dab0-45fa-b347-056928127e9d\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " Jun 11 20:58:10.130910 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.130911 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-offline-tls\") pod \"1b55b440-dab0-45fa-b347-056928127e9d\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " Jun 11 20:58:10.131148 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.130939 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58fhp\" (UniqueName: \"kubernetes.io/projected/1b55b440-dab0-45fa-b347-056928127e9d-kube-api-access-58fhp\") pod \"1b55b440-dab0-45fa-b347-056928127e9d\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " Jun 11 20:58:10.131148 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.131002 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-online-tls\") pod \"1b55b440-dab0-45fa-b347-056928127e9d\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " Jun 11 20:58:10.131148 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.131109 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55b440-dab0-45fa-b347-056928127e9d-feast-data\") pod \"1b55b440-dab0-45fa-b347-056928127e9d\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " Jun 11 20:58:10.131322 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.131197 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-registry-tls\") pod \"1b55b440-dab0-45fa-b347-056928127e9d\" (UID: \"1b55b440-dab0-45fa-b347-056928127e9d\") " Jun 11 20:58:10.131484 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.131463 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.131552 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.131489 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-ui-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.131552 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.131503 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt42b\" (UniqueName: \"kubernetes.io/projected/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-kube-api-access-rt42b\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.131552 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.131517 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.131552 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.131530 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfb4a526-5204-46b3-8457-fe7c73d2a3e8-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.131753 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.131644 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b55b440-dab0-45fa-b347-056928127e9d-feast-data" (OuterVolumeSpecName: "feast-data") pod "1b55b440-dab0-45fa-b347-056928127e9d" (UID: "1b55b440-dab0-45fa-b347-056928127e9d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:58:10.133258 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.133236 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1b55b440-dab0-45fa-b347-056928127e9d" (UID: "1b55b440-dab0-45fa-b347-056928127e9d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:58:10.133669 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.133652 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-online-tls" (OuterVolumeSpecName: "online-tls") pod "1b55b440-dab0-45fa-b347-056928127e9d" (UID: "1b55b440-dab0-45fa-b347-056928127e9d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:58:10.133743 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.133677 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1b55b440-dab0-45fa-b347-056928127e9d" (UID: "1b55b440-dab0-45fa-b347-056928127e9d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:58:10.133743 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.133677 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b55b440-dab0-45fa-b347-056928127e9d-kube-api-access-58fhp" (OuterVolumeSpecName: "kube-api-access-58fhp") pod "1b55b440-dab0-45fa-b347-056928127e9d" (UID: "1b55b440-dab0-45fa-b347-056928127e9d"). InnerVolumeSpecName "kube-api-access-58fhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:58:10.133743 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.133712 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1b55b440-dab0-45fa-b347-056928127e9d" (UID: "1b55b440-dab0-45fa-b347-056928127e9d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 11 20:58:10.232548 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.232508 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-registry-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.232548 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.232541 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-ui-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.232548 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.232549 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-offline-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.232548 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.232558 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58fhp\" (UniqueName: \"kubernetes.io/projected/1b55b440-dab0-45fa-b347-056928127e9d-kube-api-access-58fhp\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.232861 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.232567 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b55b440-dab0-45fa-b347-056928127e9d-online-tls\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.232861 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.232576 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55b440-dab0-45fa-b347-056928127e9d-feast-data\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:10.382719 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.382632 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" Jun 11 20:58:10.382719 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.382632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl" event={"ID":"bfb4a526-5204-46b3-8457-fe7c73d2a3e8","Type":"ContainerDied","Data":"78dba001fae1e410fadeb9e1daac73990a3b0bb63cdaccbd77157711b37325b9"} Jun 11 20:58:10.383240 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.382760 2564 scope.go:117] "RemoveContainer" containerID="199c9667c5b0ee674a703ea9bf9bde05f40bc3eeb5dafc2b0b2a1701ed196179" Jun 11 20:58:10.385625 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.385601 2564 generic.go:356] "Generic (PLEG): container finished" podID="1b55b440-dab0-45fa-b347-056928127e9d" containerID="50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c" exitCode=137 Jun 11 20:58:10.385625 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.385621 2564 generic.go:356] "Generic (PLEG): container finished" podID="1b55b440-dab0-45fa-b347-056928127e9d" containerID="e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0" exitCode=137 Jun 11 20:58:10.385822 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.385638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerDied","Data":"50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c"} Jun 11 20:58:10.385822 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.385666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerDied","Data":"e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0"} Jun 11 20:58:10.385822 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.385683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" event={"ID":"1b55b440-dab0-45fa-b347-056928127e9d","Type":"ContainerDied","Data":"f2a79e74fe421352b4dc906fb0d0b443ae28ff064b3cf027df4cb7042db1c7c1"} Jun 11 20:58:10.385822 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.385696 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd" Jun 11 20:58:10.393325 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.393301 2564 scope.go:117] "RemoveContainer" containerID="f1301589a1b774477e2d9bb9a3021c75958a2e5dca5fc5d3193e1c3219240982" Jun 11 20:58:10.401598 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.401576 2564 scope.go:117] "RemoveContainer" containerID="f30464c9077f6cd650600e80982bc01e798943646c11bc0e1e7792c403f81c8f" Jun 11 20:58:10.405209 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.405183 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl"] Jun 11 20:58:10.408215 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.408186 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-s8zbl"] Jun 11 20:58:10.410887 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.410871 2564 scope.go:117] "RemoveContainer" containerID="2b0209bb01ad18a0edcb67e79955df35e21be369d018c23d66ec9f88a9cdb1dd" Jun 11 20:58:10.417797 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.417771 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd"] Jun 11 20:58:10.419564 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.419543 2564 scope.go:117] "RemoveContainer" containerID="72bb6418fadbb4724cf4aca851b40cae65670d8cd960477a0bcb65d409c0f732" Jun 11 20:58:10.421369 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.421348 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-b86qd"] Jun 11 20:58:10.427738 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.427719 2564 scope.go:117] "RemoveContainer" containerID="6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd" Jun 11 20:58:10.436256 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.436237 2564 scope.go:117] "RemoveContainer" containerID="50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c" Jun 11 20:58:10.444357 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.444337 2564 scope.go:117] "RemoveContainer" containerID="1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544" Jun 11 20:58:10.452286 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.452266 2564 scope.go:117] "RemoveContainer" containerID="e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0" Jun 11 20:58:10.460415 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.460394 2564 scope.go:117] "RemoveContainer" containerID="8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768" Jun 11 20:58:10.468101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.468078 2564 scope.go:117] "RemoveContainer" containerID="0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0" Jun 11 20:58:10.475785 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.475766 2564 scope.go:117] "RemoveContainer" containerID="6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd" Jun 11 20:58:10.476107 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:10.476086 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd\": container with ID starting with 6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd not found: ID does not exist" containerID="6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd" Jun 11 20:58:10.476163 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.476120 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd"} err="failed to get container status \"6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd\": rpc error: code = NotFound desc = could not find container \"6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd\": container with ID starting with 6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd not found: ID does not exist" Jun 11 20:58:10.476163 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.476150 2564 scope.go:117] "RemoveContainer" containerID="50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c" Jun 11 20:58:10.476465 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:10.476444 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c\": container with ID starting with 50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c not found: ID does not exist" containerID="50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c" Jun 11 20:58:10.476531 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.476470 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c"} err="failed to get container status \"50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c\": rpc error: code = NotFound desc = could not find container \"50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c\": container with ID starting with 50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c not found: ID does not exist" Jun 11 20:58:10.476531 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.476487 2564 scope.go:117] "RemoveContainer" containerID="1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544" Jun 11 20:58:10.476745 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:10.476726 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544\": container with ID starting with 1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544 not found: ID does not exist" containerID="1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544" Jun 11 20:58:10.476811 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.476752 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544"} err="failed to get container status \"1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544\": rpc error: code = NotFound desc = could not find container \"1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544\": container with ID starting with 1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544 not found: ID does not exist" Jun 11 20:58:10.476811 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.476773 2564 scope.go:117] "RemoveContainer" containerID="e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0" Jun 11 20:58:10.477041 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:10.477025 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0\": container with ID starting with e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0 not found: ID does not exist" containerID="e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0" Jun 11 20:58:10.477099 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.477044 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0"} err="failed to get container status \"e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0\": rpc error: code = NotFound desc = could not find container \"e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0\": container with ID starting with e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0 not found: ID does not exist" Jun 11 20:58:10.477099 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.477058 2564 scope.go:117] "RemoveContainer" containerID="8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768" Jun 11 20:58:10.477260 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:10.477244 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768\": container with ID starting with 8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768 not found: ID does not exist" containerID="8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768" Jun 11 20:58:10.477303 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.477264 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768"} err="failed to get container status \"8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768\": rpc error: code = NotFound desc = could not find container \"8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768\": container with ID starting with 8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768 not found: ID does not exist" Jun 11 20:58:10.477303 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.477279 2564 scope.go:117] "RemoveContainer" containerID="0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0" Jun 11 20:58:10.477506 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:10.477489 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0\": container with ID starting with 0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0 not found: ID does not exist" containerID="0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0" Jun 11 20:58:10.477568 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.477513 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0"} err="failed to get container status \"0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0\": rpc error: code = NotFound desc = could not find container \"0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0\": container with ID starting with 0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0 not found: ID does not exist" Jun 11 20:58:10.477568 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.477532 2564 scope.go:117] "RemoveContainer" containerID="6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd" Jun 11 20:58:10.477764 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.477747 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd"} err="failed to get container status \"6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd\": rpc error: code = NotFound desc = could not find container \"6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd\": container with ID starting with 6ded0a66d531d1509ed30c0e4640cdc1c1eb2a9880d98c27f778b9c4e19c3afd not found: ID does not exist" Jun 11 20:58:10.477816 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.477765 2564 scope.go:117] "RemoveContainer" containerID="50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c" Jun 11 20:58:10.478059 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.478040 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c"} err="failed to get container status \"50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c\": rpc error: code = NotFound desc = could not find container \"50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c\": container with ID starting with 50d92dc91a578d40c113f491e157776795c19866765044e33b63eb4f231b444c not found: ID does not exist" Jun 11 20:58:10.478142 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.478061 2564 scope.go:117] "RemoveContainer" containerID="1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544" Jun 11 20:58:10.478281 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.478265 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544"} err="failed to get container status \"1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544\": rpc error: code = NotFound desc = could not find container \"1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544\": container with ID starting with 1290a3a54fee4b9b6a727ce6d2c2094c26c9d8f8a0e9c8446dcb06046983e544 not found: ID does not exist" Jun 11 20:58:10.478342 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.478282 2564 scope.go:117] "RemoveContainer" containerID="e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0" Jun 11 20:58:10.478512 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.478493 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0"} err="failed to get container status \"e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0\": rpc error: code = NotFound desc = could not find container \"e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0\": container with ID starting with e60f09e2ed7a4f4759feee926bc956f876f463d9322ee8a73b537056b5dac5a0 not found: ID does not exist" Jun 11 20:58:10.478562 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.478513 2564 scope.go:117] "RemoveContainer" containerID="8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768" Jun 11 20:58:10.478712 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.478694 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768"} err="failed to get container status \"8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768\": rpc error: code = NotFound desc = could not find container \"8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768\": container with ID starting with 8d9ba473ca532b882400c8d6b32b60a630b169ea67c44cb9856ee7a0e88e2768 not found: ID does not exist" Jun 11 20:58:10.478781 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.478713 2564 scope.go:117] "RemoveContainer" containerID="0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0" Jun 11 20:58:10.478922 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:10.478907 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0"} err="failed to get container status \"0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0\": rpc error: code = NotFound desc = could not find container \"0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0\": container with ID starting with 0de8fe6bf89feafe5c0111b1f1710cc1a1c9203d6a6b57a80dc09e4b0fd4caf0 not found: ID does not exist" Jun 11 20:58:11.075021 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:11.074958 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b55b440-dab0-45fa-b347-056928127e9d" path="/var/lib/kubelet/pods/1b55b440-dab0-45fa-b347-056928127e9d/volumes" Jun 11 20:58:11.075620 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:11.075607 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" path="/var/lib/kubelet/pods/bfb4a526-5204-46b3-8457-fe7c73d2a3e8/volumes" Jun 11 20:58:28.432686 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.432594 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzvmm/must-gather-tjd94"] Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.432981 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="ui" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.432998 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="ui" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433008 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="online" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433013 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="online" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433041 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="offline" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433048 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="offline" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433055 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="feast-init" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433060 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="feast-init" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433069 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="feast-apply" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433074 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="feast-apply" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433082 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="offline" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433087 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="offline" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433092 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="online" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433097 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="online" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433106 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="feast-apply" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433111 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="feast-apply" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433116 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="ui" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433121 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="ui" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433130 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="registry" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433135 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="registry" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433142 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="feast-init" Jun 11 20:58:28.433186 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433149 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="feast-init" Jun 11 20:58:28.433919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433222 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="offline" Jun 11 20:58:28.433919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433235 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="online" Jun 11 20:58:28.433919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433244 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="registry" Jun 11 20:58:28.433919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433258 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="ui" Jun 11 20:58:28.433919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433270 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="ui" Jun 11 20:58:28.433919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433279 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b55b440-dab0-45fa-b347-056928127e9d" containerName="online" Jun 11 20:58:28.433919 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.433285 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb4a526-5204-46b3-8457-fe7c73d2a3e8" containerName="offline" Jun 11 20:58:28.436341 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.436323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:28.438425 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.438400 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzvmm\"/\"kube-root-ca.crt\"" Jun 11 20:58:28.439143 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.439124 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzvmm\"/\"openshift-service-ca.crt\"" Jun 11 20:58:28.439226 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.439197 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zzvmm\"/\"default-dockercfg-dvd2k\"" Jun 11 20:58:28.443326 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.443294 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzvmm/must-gather-tjd94"] Jun 11 20:58:28.589876 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.589825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd2548ed-0975-4703-9b24-c1b64b8997b4-must-gather-output\") pod \"must-gather-tjd94\" (UID: \"bd2548ed-0975-4703-9b24-c1b64b8997b4\") " pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:28.590090 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.589949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrk4\" (UniqueName: \"kubernetes.io/projected/bd2548ed-0975-4703-9b24-c1b64b8997b4-kube-api-access-ggrk4\") pod \"must-gather-tjd94\" (UID: \"bd2548ed-0975-4703-9b24-c1b64b8997b4\") " pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:28.691455 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.691352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrk4\" (UniqueName: \"kubernetes.io/projected/bd2548ed-0975-4703-9b24-c1b64b8997b4-kube-api-access-ggrk4\") pod \"must-gather-tjd94\" (UID: \"bd2548ed-0975-4703-9b24-c1b64b8997b4\") " pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:28.691621 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.691518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd2548ed-0975-4703-9b24-c1b64b8997b4-must-gather-output\") pod \"must-gather-tjd94\" (UID: \"bd2548ed-0975-4703-9b24-c1b64b8997b4\") " pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:28.691850 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.691834 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd2548ed-0975-4703-9b24-c1b64b8997b4-must-gather-output\") pod \"must-gather-tjd94\" (UID: \"bd2548ed-0975-4703-9b24-c1b64b8997b4\") " pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:28.698592 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.698558 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrk4\" (UniqueName: \"kubernetes.io/projected/bd2548ed-0975-4703-9b24-c1b64b8997b4-kube-api-access-ggrk4\") pod \"must-gather-tjd94\" (UID: \"bd2548ed-0975-4703-9b24-c1b64b8997b4\") " pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:28.758696 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.758659 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:28.892621 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.892547 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzvmm/must-gather-tjd94"] Jun 11 20:58:28.901953 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:28.901913 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 11 20:58:29.470888 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:29.470815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzvmm/must-gather-tjd94" event={"ID":"bd2548ed-0975-4703-9b24-c1b64b8997b4","Type":"ContainerStarted","Data":"2f4046b5ad219618ac5662e2a25e965864d8a94f05fe0c27a9b77e5386a209c3"} Jun 11 20:58:35.499729 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:35.499687 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzvmm/must-gather-tjd94" event={"ID":"bd2548ed-0975-4703-9b24-c1b64b8997b4","Type":"ContainerStarted","Data":"e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc"} Jun 11 20:58:35.500187 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:35.499744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzvmm/must-gather-tjd94" event={"ID":"bd2548ed-0975-4703-9b24-c1b64b8997b4","Type":"ContainerStarted","Data":"e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295"} Jun 11 20:58:35.515018 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:35.514893 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zzvmm/must-gather-tjd94" podStartSLOduration=1.405424848 podStartE2EDuration="7.514877893s" totalImagesPullingTime="6.109453045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-11 20:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:58:35.513978505 +0000 UTC m=+1980.868724824" watchObservedRunningTime="2026-06-11 20:58:35.514877893 +0000 UTC m=+1980.869624208" Jun 11 20:58:45.544052 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:45.544017 2564 generic.go:356] "Generic (PLEG): container finished" podID="bd2548ed-0975-4703-9b24-c1b64b8997b4" containerID="e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295" exitCode=0 Jun 11 20:58:45.544486 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:45.544072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzvmm/must-gather-tjd94" event={"ID":"bd2548ed-0975-4703-9b24-c1b64b8997b4","Type":"ContainerDied","Data":"e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295"} Jun 11 20:58:45.544486 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:45.544425 2564 scope.go:117] "RemoveContainer" containerID="e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295" Jun 11 20:58:45.984387 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:45.984301 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zzvmm_must-gather-tjd94_bd2548ed-0975-4703-9b24-c1b64b8997b4/gather/0.log" Jun 11 20:58:48.962611 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:48.962584 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v8w2z_b0a18e54-07ff-4255-b8b9-998883464104/global-pull-secret-syncer/0.log" Jun 11 20:58:49.053293 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:49.053258 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r28zn_1e66429f-dbc0-452f-9fe8-795b63558fd3/konnectivity-agent/0.log" Jun 11 20:58:49.091620 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:49.091590 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-254.ec2.internal_99904d988681166720269c6ecc3484aa/haproxy/0.log" Jun 11 20:58:51.303224 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.303192 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zzvmm/must-gather-tjd94"] Jun 11 20:58:51.303819 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.303417 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-zzvmm/must-gather-tjd94" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" containerName="copy" containerID="cri-o://e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc" gracePeriod=2 Jun 11 20:58:51.305284 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:51.305255 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tjd94\" is forbidden: User \"system:node:ip-10-0-137-254.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzvmm\": no relationship found between node 'ip-10-0-137-254.ec2.internal' and this object" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:51.305464 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.305427 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zzvmm/must-gather-tjd94"] Jun 11 20:58:51.533587 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.533557 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zzvmm_must-gather-tjd94_bd2548ed-0975-4703-9b24-c1b64b8997b4/copy/0.log" Jun 11 20:58:51.535216 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.535196 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:51.537086 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:51.537064 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tjd94\" is forbidden: User \"system:node:ip-10-0-137-254.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzvmm\": no relationship found between node 'ip-10-0-137-254.ec2.internal' and this object" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:51.565659 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.565575 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd2548ed-0975-4703-9b24-c1b64b8997b4-must-gather-output\") pod \"bd2548ed-0975-4703-9b24-c1b64b8997b4\" (UID: \"bd2548ed-0975-4703-9b24-c1b64b8997b4\") " Jun 11 20:58:51.565659 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.565622 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggrk4\" (UniqueName: \"kubernetes.io/projected/bd2548ed-0975-4703-9b24-c1b64b8997b4-kube-api-access-ggrk4\") pod \"bd2548ed-0975-4703-9b24-c1b64b8997b4\" (UID: \"bd2548ed-0975-4703-9b24-c1b64b8997b4\") " Jun 11 20:58:51.566028 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.565995 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2548ed-0975-4703-9b24-c1b64b8997b4-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "bd2548ed-0975-4703-9b24-c1b64b8997b4" (UID: "bd2548ed-0975-4703-9b24-c1b64b8997b4"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 11 20:58:51.567932 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.567893 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2548ed-0975-4703-9b24-c1b64b8997b4-kube-api-access-ggrk4" (OuterVolumeSpecName: "kube-api-access-ggrk4") pod "bd2548ed-0975-4703-9b24-c1b64b8997b4" (UID: "bd2548ed-0975-4703-9b24-c1b64b8997b4"). InnerVolumeSpecName "kube-api-access-ggrk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 11 20:58:51.569600 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.569581 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zzvmm_must-gather-tjd94_bd2548ed-0975-4703-9b24-c1b64b8997b4/copy/0.log" Jun 11 20:58:51.569958 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.569932 2564 generic.go:356] "Generic (PLEG): container finished" podID="bd2548ed-0975-4703-9b24-c1b64b8997b4" containerID="e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc" exitCode=143 Jun 11 20:58:51.570085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.569985 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:51.570085 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.570022 2564 scope.go:117] "RemoveContainer" containerID="e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc" Jun 11 20:58:51.571982 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:51.571927 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tjd94\" is forbidden: User \"system:node:ip-10-0-137-254.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzvmm\": no relationship found between node 'ip-10-0-137-254.ec2.internal' and this object" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:51.580320 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.580293 2564 scope.go:117] "RemoveContainer" containerID="e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295" Jun 11 20:58:51.582593 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:51.582565 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tjd94\" is forbidden: User \"system:node:ip-10-0-137-254.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zzvmm\": no relationship found between node 'ip-10-0-137-254.ec2.internal' and this object" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" pod="openshift-must-gather-zzvmm/must-gather-tjd94" Jun 11 20:58:51.591881 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.591861 2564 scope.go:117] "RemoveContainer" containerID="e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc" Jun 11 20:58:51.592175 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:51.592150 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc\": container with ID starting with e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc not found: ID does not exist" containerID="e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc" Jun 11 20:58:51.592265 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.592186 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc"} err="failed to get container status \"e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc\": rpc error: code = NotFound desc = could not find container \"e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc\": container with ID starting with e992fc028aaee241cd7c92c288a74adb94cec425a8bce9e2d6b4ef1efe93bdcc not found: ID does not exist" Jun 11 20:58:51.592265 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.592211 2564 scope.go:117] "RemoveContainer" containerID="e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295" Jun 11 20:58:51.592463 ip-10-0-137-254 kubenswrapper[2564]: E0611 20:58:51.592446 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295\": container with ID starting with e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295 not found: ID does not exist" containerID="e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295" Jun 11 20:58:51.592504 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.592469 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295"} err="failed to get container status \"e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295\": rpc error: code = NotFound desc = could not find container \"e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295\": container with ID starting with e1a75f35cfb0a33d76fb181766068604ed970da43a07920a450d9e7b9f9dd295 not found: ID does not exist" Jun 11 20:58:51.667013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.666953 2564 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd2548ed-0975-4703-9b24-c1b64b8997b4-must-gather-output\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:51.667013 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:51.667005 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrk4\" (UniqueName: \"kubernetes.io/projected/bd2548ed-0975-4703-9b24-c1b64b8997b4-kube-api-access-ggrk4\") on node \"ip-10-0-137-254.ec2.internal\" DevicePath \"\"" Jun 11 20:58:52.826289 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:52.826257 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84b49b6d67-9dvrm_24a55637-1ee5-4825-9a7b-2cae9e594ac5/metrics-server/0.log" Jun 11 20:58:52.959607 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:52.959551 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkqtr_933a7689-fe60-4573-ba8c-ec03a820295f/node-exporter/0.log" Jun 11 20:58:52.982233 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:52.982197 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkqtr_933a7689-fe60-4573-ba8c-ec03a820295f/kube-rbac-proxy/0.log" Jun 11 20:58:53.003029 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.003000 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkqtr_933a7689-fe60-4573-ba8c-ec03a820295f/init-textfile/0.log" Jun 11 20:58:53.077819 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.077735 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" path="/var/lib/kubelet/pods/bd2548ed-0975-4703-9b24-c1b64b8997b4/volumes" Jun 11 20:58:53.099064 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.099030 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-bjkkp_c70e158e-548b-44a4-969d-d701a609aa7f/kube-rbac-proxy-main/0.log" Jun 11 20:58:53.118126 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.118098 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-bjkkp_c70e158e-548b-44a4-969d-d701a609aa7f/kube-rbac-proxy-self/0.log" Jun 11 20:58:53.137556 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.137523 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-bjkkp_c70e158e-548b-44a4-969d-d701a609aa7f/openshift-state-metrics/0.log" Jun 11 20:58:53.379254 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.379175 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5cf667749c-krq5h_5f55f499-98f4-46dd-b574-9dbaf61c8c3a/prometheus-operator-admission-webhook/0.log" Jun 11 20:58:53.478050 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.478017 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/thanos-query/0.log" Jun 11 20:58:53.499102 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.499077 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/kube-rbac-proxy-web/0.log" Jun 11 20:58:53.520377 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.520349 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/kube-rbac-proxy/0.log" Jun 11 20:58:53.542564 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.542538 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/prom-label-proxy/0.log" Jun 11 20:58:53.562680 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.562652 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/kube-rbac-proxy-rules/0.log" Jun 11 20:58:53.584084 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:53.584054 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c8cffbc7-8jq9m_a96f5138-1a26-4752-a96c-414575e9d99f/kube-rbac-proxy-metrics/0.log" Jun 11 20:58:54.845877 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:54.845851 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-867b76cb74-dt22k_8a2c57c0-8141-4960-8b86-d8783a208d7e/networking-console-plugin/0.log" Jun 11 20:58:55.318655 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:55.318621 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/1.log" Jun 11 20:58:55.324948 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:55.324921 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5dbbb5d744-kf9lp_8aaeb72a-cce3-4f1c-a111-69d9c132d38e/console-operator/2.log" Jun 11 20:58:56.223368 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.223337 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-bd4ccd799-nk2zj_d12a2422-a9e9-42f2-b5fe-008e194c6a0d/volume-data-source-validator/0.log" Jun 11 20:58:56.544795 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.544761 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z"] Jun 11 20:58:56.545110 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.545096 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" containerName="gather" Jun 11 20:58:56.545163 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.545111 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" containerName="gather" Jun 11 20:58:56.545163 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.545124 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" containerName="copy" Jun 11 20:58:56.545163 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.545131 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" containerName="copy" Jun 11 20:58:56.545260 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.545183 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" containerName="gather" Jun 11 20:58:56.545260 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.545190 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2548ed-0975-4703-9b24-c1b64b8997b4" containerName="copy" Jun 11 20:58:56.548185 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.548167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.550074 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.550050 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hskms\"/\"default-dockercfg-vqcww\"" Jun 11 20:58:56.550182 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.550111 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hskms\"/\"openshift-service-ca.crt\"" Jun 11 20:58:56.550759 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.550739 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hskms\"/\"kube-root-ca.crt\"" Jun 11 20:58:56.553639 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.553616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z"] Jun 11 20:58:56.611279 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.611245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-sys\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.611487 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.611285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-lib-modules\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.611487 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.611358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-podres\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.611487 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.611408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-proc\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.611487 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.611427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5rw\" (UniqueName: \"kubernetes.io/projected/7a07e176-84f4-4431-8f37-0311094fe6b6-kube-api-access-8m5rw\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.712735 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.712701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-podres\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.712735 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.712752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-proc\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.713039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.712770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5rw\" (UniqueName: \"kubernetes.io/projected/7a07e176-84f4-4431-8f37-0311094fe6b6-kube-api-access-8m5rw\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.713039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.712851 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-proc\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.713039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.712897 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-podres\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.713039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.712905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-sys\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.713039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.712948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-lib-modules\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.713039 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.713020 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-sys\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.713237 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.713079 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a07e176-84f4-4431-8f37-0311094fe6b6-lib-modules\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.720195 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.720170 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5rw\" (UniqueName: \"kubernetes.io/projected/7a07e176-84f4-4431-8f37-0311094fe6b6-kube-api-access-8m5rw\") pod \"perf-node-gather-daemonset-4n68z\" (UID: \"7a07e176-84f4-4431-8f37-0311094fe6b6\") " pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.859401 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.859302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:56.985260 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:56.985235 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z"] Jun 11 20:58:56.988173 ip-10-0-137-254 kubenswrapper[2564]: W0611 20:58:56.988134 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a07e176_84f4_4431_8f37_0311094fe6b6.slice/crio-c26634f31a7e7b66f19f168c2d117f89a776557503893e7904c234b72839212e WatchSource:0}: Error finding container c26634f31a7e7b66f19f168c2d117f89a776557503893e7904c234b72839212e: Status 404 returned error can't find the container with id c26634f31a7e7b66f19f168c2d117f89a776557503893e7904c234b72839212e Jun 11 20:58:57.084147 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:57.084120 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qpm6t_e98f77d6-3aa7-4ace-8987-93b443679091/dns/0.log" Jun 11 20:58:57.101690 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:57.101665 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qpm6t_e98f77d6-3aa7-4ace-8987-93b443679091/kube-rbac-proxy/0.log" Jun 11 20:58:57.123040 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:57.123002 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5fnqb_f748b10a-2034-4330-9292-f4781f1b4359/dns-node-resolver/0.log" Jun 11 20:58:57.551151 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:57.551123 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k58bv_c0e098fe-1a43-4895-a7e0-ab36688d5b77/node-ca/0.log" Jun 11 20:58:57.595791 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:57.595750 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" event={"ID":"7a07e176-84f4-4431-8f37-0311094fe6b6","Type":"ContainerStarted","Data":"0fb9b152bead8c489e462c4578b981a5a7048ad205b5b7209d2a92aa8fab1529"} Jun 11 20:58:57.596018 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:57.595812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" event={"ID":"7a07e176-84f4-4431-8f37-0311094fe6b6","Type":"ContainerStarted","Data":"c26634f31a7e7b66f19f168c2d117f89a776557503893e7904c234b72839212e"} Jun 11 20:58:57.596018 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:57.595894 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:58:57.609410 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:57.609352 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" podStartSLOduration=1.6093377690000001 podStartE2EDuration="1.609337769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-11 20:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-11 20:58:57.608799865 +0000 UTC m=+2002.963546183" watchObservedRunningTime="2026-06-11 20:58:57.609337769 +0000 UTC m=+2002.964084084" Jun 11 20:58:58.195417 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:58.195383 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76bc48fcc4-kpmwf_fb12d4fb-c613-4500-9bf5-f09a5f001dc4/router/0.log" Jun 11 20:58:58.505526 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:58.505488 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fgbx2_01568624-0240-48ae-9f43-2bc8eb597dcd/serve-healthcheck-canary/0.log" Jun 11 20:58:58.860607 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:58.860488 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5bbd86d6bd-z825d_c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f/insights-operator/0.log" Jun 11 20:58:58.862749 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:58.862722 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5bbd86d6bd-z825d_c7fa4480-6eee-4529-9ffc-c7b7b4b17c8f/insights-operator/1.log" Jun 11 20:58:58.885417 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:58.885390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cq7zt_abbfb6e9-d45b-418d-8691-9b5cbb123d13/kube-rbac-proxy/0.log" Jun 11 20:58:58.902997 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:58.902949 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cq7zt_abbfb6e9-d45b-418d-8691-9b5cbb123d13/exporter/0.log" Jun 11 20:58:58.920412 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:58:58.920373 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cq7zt_abbfb6e9-d45b-418d-8691-9b5cbb123d13/extractor/0.log" Jun 11 20:59:02.926278 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:02.926240 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7c578b78fc-lxq24_98700412-c4d4-4989-8edf-90b8e43cf90d/kube-storage-version-migrator-operator/1.log" Jun 11 20:59:02.926925 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:02.926903 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7c578b78fc-lxq24_98700412-c4d4-4989-8edf-90b8e43cf90d/kube-storage-version-migrator-operator/0.log" Jun 11 20:59:03.610163 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:03.610133 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hskms/perf-node-gather-daemonset-4n68z" Jun 11 20:59:03.649488 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:03.649456 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97sxc_9f84094c-d325-462d-8f1b-a71c48ad5524/kube-multus-additional-cni-plugins/0.log" Jun 11 20:59:03.667212 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:03.667184 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97sxc_9f84094c-d325-462d-8f1b-a71c48ad5524/egress-router-binary-copy/0.log" Jun 11 20:59:03.684616 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:03.684592 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97sxc_9f84094c-d325-462d-8f1b-a71c48ad5524/cni-plugins/0.log" Jun 11 20:59:03.701467 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:03.701445 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97sxc_9f84094c-d325-462d-8f1b-a71c48ad5524/bond-cni-plugin/0.log" Jun 11 20:59:03.718211 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:03.718186 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97sxc_9f84094c-d325-462d-8f1b-a71c48ad5524/routeoverride-cni/0.log" Jun 11 20:59:03.735241 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:03.735210 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97sxc_9f84094c-d325-462d-8f1b-a71c48ad5524/whereabouts-cni-bincopy/0.log" Jun 11 20:59:03.751798 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:03.751767 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97sxc_9f84094c-d325-462d-8f1b-a71c48ad5524/whereabouts-cni/0.log" Jun 11 20:59:04.084993 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:04.084949 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v8xfv_69a4d19c-0866-442f-af12-6bd23fc4c17e/kube-multus/0.log" Jun 11 20:59:04.185197 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:04.185152 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wqxvp_86b6d922-1a55-4844-b308-127fc062e056/network-metrics-daemon/0.log" Jun 11 20:59:04.201760 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:04.201736 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wqxvp_86b6d922-1a55-4844-b308-127fc062e056/kube-rbac-proxy/0.log" Jun 11 20:59:04.910779 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:04.910754 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-controller/0.log" Jun 11 20:59:04.929354 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:04.929319 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/1.log" Jun 11 20:59:04.929521 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:04.929367 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovn-acl-logging/0.log" Jun 11 20:59:04.947217 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:04.947176 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/kube-rbac-proxy-node/0.log" Jun 11 20:59:04.966194 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:04.966153 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/kube-rbac-proxy-ovn-metrics/0.log" Jun 11 20:59:04.983043 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:04.983017 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/northd/0.log" Jun 11 20:59:05.000453 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:05.000421 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/nbdb/0.log" Jun 11 20:59:05.018145 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:05.018113 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/sbdb/0.log" Jun 11 20:59:05.060562 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:05.060529 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5z7f_a715e5d9-2205-4873-b97a-6e9a53d9acca/ovnkube-controller/0.log" Jun 11 20:59:06.460916 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:06.460885 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jv82t_24fe286b-e2a1-4004-87d8-450503f21c90/network-check-target-container/0.log" Jun 11 20:59:07.344430 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:07.344403 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-fk57z_942c56c3-9856-4d26-82a4-f9d7c1adc1f9/iptables-alerter/0.log" Jun 11 20:59:07.947083 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:07.947003 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-qw9mb_f085aed6-890e-4ac3-9129-6fc81f1f3d44/tuned/0.log" Jun 11 20:59:09.487359 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:09.487330 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-9547488fd-j8xrc_344687ec-6871-4883-b290-c89548faa703/cluster-samples-operator/0.log" Jun 11 20:59:09.501358 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:09.501324 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-9547488fd-j8xrc_344687ec-6871-4883-b290-c89548faa703/cluster-samples-operator-watch/0.log" Jun 11 20:59:10.361248 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:10.361217 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-7ccbfd59dc-mf44h_8bf7b5a5-bb18-47ff-909c-6c3201f88856/service-ca-operator/1.log" Jun 11 20:59:10.362106 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:10.362087 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-7ccbfd59dc-mf44h_8bf7b5a5-bb18-47ff-909c-6c3201f88856/service-ca-operator/0.log" Jun 11 20:59:10.626756 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:10.626672 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-7b994bc59-8ct6j_ec9dfe77-2d1f-4b8b-9e28-5c145c6da331/service-ca-controller/0.log" Jun 11 20:59:11.018870 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:11.018834 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wmwk6_d3aec8fe-cd66-48f3-970b-0e65561d692b/csi-driver/0.log" Jun 11 20:59:11.036101 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:11.036065 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wmwk6_d3aec8fe-cd66-48f3-970b-0e65561d692b/csi-node-driver-registrar/0.log" Jun 11 20:59:11.052767 ip-10-0-137-254 kubenswrapper[2564]: I0611 20:59:11.052736 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wmwk6_d3aec8fe-cd66-48f3-970b-0e65561d692b/csi-liveness-probe/0.log"