Jul 16 17:30:19.201140 ip-10-0-131-83 systemd[1]: Starting Kubernetes Kubelet... Jul 16 17:30:19.649151 ip-10-0-131-83 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:30:19.649151 ip-10-0-131-83 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 17:30:19.649151 ip-10-0-131-83 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:30:19.649151 ip-10-0-131-83 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 17:30:19.649151 ip-10-0-131-83 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:30:19.650196 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.650110 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 17:30:19.655486 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655465 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:30:19.655486 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655483 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:30:19.655486 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655487 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:30:19.655486 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655490 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:30:19.655486 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655493 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655498 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655501 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655504 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655507 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655509 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655512 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655515 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655517 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655520 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655523 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655525 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655528 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655531 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655533 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655536 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655538 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655540 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655543 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655545 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655551 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:30:19.655673 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655553 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655556 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655559 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655561 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655564 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655566 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655569 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655571 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655573 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655576 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655578 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655581 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655584 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655587 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655590 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655592 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655595 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655597 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655600 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655602 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:30:19.656184 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655605 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655607 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655610 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655612 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655615 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655617 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655620 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655623 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655625 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655628 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655631 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655633 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655636 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655640 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655644 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655647 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655650 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655652 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655655 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:30:19.656670 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655657 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655660 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655663 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655666 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655669 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655671 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655675 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655678 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655681 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655684 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655686 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655689 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655692 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655695 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655700 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655702 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655705 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655707 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655710 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655715 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:30:19.657138 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655718 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655720 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655723 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655727 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655731 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655734 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655738 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655741 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655743 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655746 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655748 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655751 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655754 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655756 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:30:19.657638 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.655758 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:30:19.658055 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658042 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:30:19.658055 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658055 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658058 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658062 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658065 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658067 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658070 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658073 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658076 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658079 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658081 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658084 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658086 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658089 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658091 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658093 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658096 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658099 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658101 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658105 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658107 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:30:19.658109 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658110 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658113 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658116 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658118 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658121 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658124 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658127 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658129 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658132 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658135 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658137 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658140 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658143 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658145 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658149 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658153 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658156 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658158 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658161 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658163 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:30:19.658590 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658166 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658168 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658171 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658173 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658176 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658178 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658181 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658183 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658188 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658192 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658195 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658198 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658201 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658204 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658207 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658210 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658213 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658215 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658218 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658220 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:30:19.659101 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658223 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658225 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658228 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658230 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658233 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658236 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658238 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658241 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658244 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658247 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658251 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658254 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658257 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658259 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658262 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658265 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658267 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658271 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658274 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:30:19.659593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658276 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658279 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658281 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658285 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658289 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658291 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658294 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658298 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658301 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658304 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658306 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658308 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658311 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658313 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658316 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658319 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658321 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658324 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658326 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658696 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:30:19.660078 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658701 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658704 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658706 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658709 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658711 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658714 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658717 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658719 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658722 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658725 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658727 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658730 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658732 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658735 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658737 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658740 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658743 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658745 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658748 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658751 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:30:19.660556 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658754 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658757 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658759 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658762 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658764 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658767 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658770 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658772 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658775 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658777 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658780 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658782 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658785 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658788 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658790 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658794 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658797 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658799 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658802 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658804 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:30:19.661048 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658807 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658809 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658812 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658814 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658816 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658819 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658822 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658824 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658828 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658830 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658851 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658858 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658862 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658866 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658868 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658871 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658874 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658877 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658880 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658882 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:30:19.661523 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658885 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658888 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658890 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658893 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658895 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658898 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658900 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658903 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658905 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658907 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658910 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658912 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658914 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658917 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658919 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658922 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658926 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658929 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658931 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658933 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:30:19.662029 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658936 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658939 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658941 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658945 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658947 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658950 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658952 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658956 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658959 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658962 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658965 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658968 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658970 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658973 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658975 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658977 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658980 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.658982 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659050 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659057 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 17:30:19.662527 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659085 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659090 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659094 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659097 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659101 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659106 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659109 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659112 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659115 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659119 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659122 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659125 2565 flags.go:64] FLAG: --cgroup-root="" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659127 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659130 2565 flags.go:64] FLAG: --client-ca-file="" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659134 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659138 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659142 2565 flags.go:64] FLAG: --cluster-domain="" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659145 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659148 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659151 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659155 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659158 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659161 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 17:30:19.663028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659164 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659167 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659177 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659181 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659185 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659188 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659191 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659195 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659198 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659201 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659204 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659206 2565 flags.go:64] FLAG: --enable-server="true" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659209 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659214 2565 flags.go:64] FLAG: --event-burst="100" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659217 2565 flags.go:64] FLAG: --event-qps="50" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659220 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659223 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659226 2565 flags.go:64] FLAG: --eviction-hard="" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659229 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659232 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659235 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659238 2565 flags.go:64] FLAG: --eviction-soft="" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659240 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659243 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659247 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 17:30:19.663591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659250 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659252 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659255 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659258 2565 flags.go:64] FLAG: --feature-gates="" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659262 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659265 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659268 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659271 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659274 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659276 2565 flags.go:64] FLAG: --help="false" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659279 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-131-83.ec2.internal" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659282 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659285 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659288 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659291 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659294 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659297 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659299 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659302 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659305 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659308 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659310 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659313 2565 flags.go:64] FLAG: --kube-reserved="" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659316 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 17:30:19.664218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659319 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659321 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659324 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659327 2565 flags.go:64] FLAG: --lock-file="" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659329 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659332 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659334 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659346 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659350 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659352 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659355 2565 flags.go:64] FLAG: --logging-format="text" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659358 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659361 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659364 2565 flags.go:64] FLAG: --manifest-url="" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659367 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659371 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659374 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659378 2565 flags.go:64] FLAG: --max-pods="110" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659380 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659383 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659386 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659389 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659392 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659395 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659397 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 17:30:19.664903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659404 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659407 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659409 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659412 2565 flags.go:64] FLAG: --pod-cidr="" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659415 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659420 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659422 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659425 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659428 2565 flags.go:64] FLAG: --port="10250" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659431 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659434 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d346cd9c71fe50f4" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659437 2565 flags.go:64] FLAG: --qos-reserved="" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659440 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659442 2565 flags.go:64] FLAG: --register-node="true" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659446 2565 flags.go:64] FLAG: --register-with-taints="" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659450 2565 flags.go:64] FLAG: --registry-burst="10" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659453 2565 flags.go:64] FLAG: --registry-qps="5" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659456 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659459 2565 flags.go:64] FLAG: --reserved-memory="" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659462 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659465 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659468 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659470 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659473 2565 flags.go:64] FLAG: --runonce="false" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659476 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 17:30:19.665503 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659478 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659481 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659484 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659487 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659490 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659493 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659495 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659498 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659501 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659503 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659506 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659509 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659511 2565 flags.go:64] FLAG: --system-cgroups="" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659514 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659519 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659521 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659524 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659528 2565 flags.go:64] FLAG: --tls-min-version="" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659530 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659533 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659536 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659540 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659543 2565 flags.go:64] FLAG: --v="2" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659547 2565 flags.go:64] FLAG: --version="false" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659551 2565 flags.go:64] FLAG: --vmodule="" Jul 16 17:30:19.666159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659555 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.659559 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659670 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659674 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659678 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659682 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659685 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659688 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659691 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659695 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659698 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659701 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659703 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659706 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659708 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659711 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659713 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659716 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659718 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:30:19.666775 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659720 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659723 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659725 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659728 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659730 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659732 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659735 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659739 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659741 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659745 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659747 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659750 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659752 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659755 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659757 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659760 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659762 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659765 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659767 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:30:19.667285 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659771 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659774 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659777 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659780 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659783 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659786 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659788 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659791 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659794 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659796 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659798 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659801 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659804 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659806 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659808 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659811 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659813 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659816 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659818 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659820 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:30:19.667757 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659824 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659827 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659831 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659834 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659852 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659855 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659858 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659861 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659864 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659867 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659869 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659871 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659874 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659876 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659879 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659881 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659884 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659886 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659889 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659891 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:30:19.668255 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659894 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659896 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659899 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659901 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659904 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659906 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659909 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659911 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659913 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659916 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659918 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659921 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659925 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659927 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659931 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659938 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659941 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659944 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659946 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659949 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:30:19.668728 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659952 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659954 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.659957 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.660673 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.668235 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.668251 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.668272 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.668281 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668334 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668338 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668342 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668344 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668347 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668350 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668353 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:30:19.669235 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668355 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668358 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668361 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668363 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668365 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668368 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668370 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668373 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668375 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668377 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668380 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668382 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668385 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668387 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668390 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668393 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668395 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668398 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668400 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668402 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:30:19.669605 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668405 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668408 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668411 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668414 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668417 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668419 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668422 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668426 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668430 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668433 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668436 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668439 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668441 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668444 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668447 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668450 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668452 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668455 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668458 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:30:19.670120 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668461 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668464 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668467 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668470 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668472 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668475 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668477 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668480 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668482 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668485 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668488 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668491 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668493 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668497 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668500 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668504 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668507 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668509 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668512 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:30:19.670587 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668515 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668517 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668520 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668522 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668525 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668527 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668529 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668532 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668534 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668537 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668540 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668542 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668544 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668547 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668549 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668551 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668554 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668557 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668559 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668562 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:30:19.671053 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668564 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668567 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668569 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668573 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668576 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668579 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668582 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668585 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668588 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668591 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668594 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668596 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668599 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668601 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.668606 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:30:19.671549 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668714 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668718 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668721 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668723 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668726 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668728 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668731 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668734 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668738 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668741 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668744 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668747 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668749 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668752 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668755 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668757 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668759 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668762 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668764 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:30:19.671946 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668767 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668769 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668771 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668774 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668776 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668779 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668781 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668784 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668786 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668789 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668791 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668794 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668796 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668799 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668801 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668803 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668806 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668808 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668811 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668813 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:30:19.672409 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668815 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668818 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668820 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668823 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668825 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668827 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668830 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668832 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668851 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668856 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668858 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668861 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668863 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668866 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668868 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668871 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668873 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668877 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668881 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:30:19.672903 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668883 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668886 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668888 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668892 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668895 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668897 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668900 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668902 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668905 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668907 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668910 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668912 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668914 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668917 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668919 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668922 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668924 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668926 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668929 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668931 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:30:19.673363 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668934 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668937 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668939 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668942 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668944 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668947 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668949 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668951 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668955 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668958 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668961 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668964 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668966 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668968 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668971 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668973 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668976 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668979 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668981 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668983 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:30:19.674072 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:19.668986 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:30:19.674554 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.668991 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:30:19.674554 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.669092 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 17:30:19.674554 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.671541 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 17:30:19.674554 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.673321 2565 server.go:1019] "Starting client certificate rotation" Jul 16 17:30:19.674554 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.673416 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:30:19.674554 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.673450 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:30:19.676238 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.676224 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:30:19.681748 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.681726 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:30:19.696474 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.696451 2565 log.go:25] "Validated CRI v1 runtime API" Jul 16 17:30:19.698934 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.698911 2565 log.go:25] "Validated CRI v1 image API" Jul 16 17:30:19.701802 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.701787 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 17:30:19.704419 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.704397 2565 fs.go:135] Filesystem UUIDs: map[3df4b433-d3c8-401d-8587-14fda08ac2e2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 81bcd498-cf65-4a73-a692-7c8d126c4242:/dev/nvme0n1p4] Jul 16 17:30:19.704500 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.704417 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 17:30:19.706324 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.706307 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:30:19.711406 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.711302 2565 manager.go:216] Machine: {Timestamp:2026-07-16 17:30:19.708854425 +0000 UTC m=+0.380324622 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3040917 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a37dd147cf699919a15980f139ffb SystemUUID:ec2a37dd-147c-f699-919a-15980f139ffb BootID:bc64380f-0c80-4392-8e87-d4186a9114d2 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ad:e2:7d:9f:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ad:e2:7d:9f:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:5b:09:c2:6d:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 17:30:19.711406 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.711400 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 17:30:19.711519 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.711480 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 17:30:19.715309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.715285 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 17:30:19.715446 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.715314 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-83.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 17:30:19.716152 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.716141 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 17:30:19.716215 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.716154 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 17:30:19.716215 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.716162 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:30:19.717030 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.717020 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:30:19.718921 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.718911 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 17:30:19.723620 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.723608 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:30:19.723738 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.723729 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 17:30:19.723771 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.723760 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 16 17:30:19.723771 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.723769 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 17:30:19.723825 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.723780 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 17:30:19.723825 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.723790 2565 kubelet.go:398] "Adding apiserver pod source" Jul 16 17:30:19.723825 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.723799 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 17:30:19.724187 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.724176 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:30:19.724243 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.724196 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:30:19.728216 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.728200 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 17:30:19.730520 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.730504 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 17:30:19.730580 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.730538 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 17:30:19.730699 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.730689 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 17:30:19.730734 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.730707 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 17:30:19.730734 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.730715 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 17:30:19.730734 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.730724 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 17:30:19.730815 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.730735 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 17:30:19.730815 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.730741 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 17:30:19.731442 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.731432 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 17:30:19.731483 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.731455 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 17:30:19.731483 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.731462 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 17:30:19.731483 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.731472 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 17:30:19.731555 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.731485 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 17:30:19.731555 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.731493 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 17:30:19.733319 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.733307 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 17:30:19.733319 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.733317 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 17:30:19.735690 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.735658 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 17:30:19.735757 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.735657 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-83.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 17:30:19.735757 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.735703 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 17:30:19.736508 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.736494 2565 server.go:1268] "Started kubelet" Jul 16 17:30:19.736612 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.736591 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 17:30:19.736721 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.736678 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 17:30:19.736789 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.736770 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 17:30:19.737480 ip-10-0-131-83 systemd[1]: Started Kubernetes Kubelet. Jul 16 17:30:19.741546 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.741107 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 17:30:19.742817 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.742782 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 16 17:30:19.743643 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.743600 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 17:30:19.748583 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.747448 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-83.ec2.internal.18c2d5621b778ca3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-83.ec2.internal,UID:ip-10-0-131-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-83.ec2.internal,},FirstTimestamp:2026-07-16 17:30:19.736468643 +0000 UTC m=+0.407938840,LastTimestamp:2026-07-16 17:30:19.736468643 +0000 UTC m=+0.407938840,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-83.ec2.internal,}" Jul 16 17:30:19.750000 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.749981 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 17:30:19.750093 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.749987 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 17:30:19.750152 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.750098 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:19.750809 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.750772 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 17:30:19.750809 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.750774 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 17:30:19.750946 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.750812 2565 factory.go:55] Registering systemd factory Jul 16 17:30:19.750946 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.750821 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 17:30:19.750946 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.750868 2565 factory.go:223] Registration of the systemd container factory successfully Jul 16 17:30:19.750946 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.750942 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 16 17:30:19.751117 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.750953 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 16 17:30:19.751117 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.751082 2565 factory.go:153] Registering CRI-O factory Jul 16 17:30:19.751117 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.751098 2565 factory.go:223] Registration of the crio container factory successfully Jul 16 17:30:19.751264 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.751150 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 17:30:19.751264 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.751180 2565 factory.go:103] Registering Raw factory Jul 16 17:30:19.751264 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.751192 2565 manager.go:1196] Started watching for new ooms in manager Jul 16 17:30:19.752279 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.752265 2565 manager.go:349] Starting recovery of all containers Jul 16 17:30:19.759053 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.759022 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 17:30:19.761637 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.761482 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 17:30:19.761637 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.761495 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 17:30:19.762202 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.762182 2565 manager.go:354] Recovery completed Jul 16 17:30:19.763539 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.763520 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tv9p6" Jul 16 17:30:19.767399 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.767386 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:30:19.769201 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.769183 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:30:19.769304 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.769219 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:30:19.769304 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.769232 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:30:19.769829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.769813 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 17:30:19.769829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.769830 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 17:30:19.769983 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.769865 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:30:19.769983 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.769909 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:30:19.770642 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.770628 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:30:19.770704 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.770655 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:30:19.770704 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.770669 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:30:19.772088 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.772077 2565 policy_none.go:49] "None policy: Start" Jul 16 17:30:19.772138 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.772092 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 17:30:19.772138 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.772103 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 17:30:19.773511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.773493 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tv9p6" Jul 16 17:30:19.774045 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.774034 2565 policy_none.go:47] "Start" Jul 16 17:30:19.778507 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.778425 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-83.ec2.internal.18c2d5621d6b0685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-83.ec2.internal,UID:ip-10-0-131-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-83.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-83.ec2.internal,},FirstTimestamp:2026-07-16 17:30:19.769202309 +0000 UTC m=+0.440672513,LastTimestamp:2026-07-16 17:30:19.769202309 +0000 UTC m=+0.440672513,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-83.ec2.internal,}" Jul 16 17:30:19.808457 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.808440 2565 manager.go:337] "Starting Device Plugin manager" Jul 16 17:30:19.808541 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.808472 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 17:30:19.808541 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.808482 2565 server.go:87] "Starting device plugin registration server" Jul 16 17:30:19.808746 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.808732 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 17:30:19.808800 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.808747 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 17:30:19.808864 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.808832 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 17:30:19.808942 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.808930 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 17:30:19.808997 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.808943 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 17:30:19.809469 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.809451 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 17:30:19.809542 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.809502 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:19.870975 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.870949 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 17:30:19.873124 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.870993 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 17:30:19.873124 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.871013 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 17:30:19.873124 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.871044 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 17:30:19.873682 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.873666 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:30:19.909637 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.909579 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:30:19.910796 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.910781 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:30:19.910908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.910812 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:30:19.910908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.910829 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:30:19.910908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.910879 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-83.ec2.internal" Jul 16 17:30:19.923691 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.923672 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-83.ec2.internal" Jul 16 17:30:19.923734 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.923695 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-83.ec2.internal\": node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:19.956697 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:19.956671 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:19.971501 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.971481 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal"] Jul 16 17:30:19.971551 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.971541 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:30:19.972413 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.972399 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:30:19.972481 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.972427 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:30:19.972481 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.972442 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:30:19.974750 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.974738 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:30:19.974867 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.974853 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 16 17:30:19.974906 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.974883 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:30:19.975430 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.975419 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:30:19.975530 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.975438 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:30:19.975530 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.975449 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:30:19.975530 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.975483 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:30:19.975530 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.975506 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:30:19.975530 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.975516 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:30:19.977663 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.977648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 16 17:30:19.977723 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.977682 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:30:19.978387 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.978368 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:30:19.978453 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.978390 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:30:19.978453 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:19.978401 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:30:20.000390 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.000369 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-83.ec2.internal\" not found" node="ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.003920 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.003902 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-83.ec2.internal\" not found" node="ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.052668 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.052631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e55879116702e2efd352cb1d1c01b0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"9e55879116702e2efd352cb1d1c01b0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.052780 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.052672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e55879116702e2efd352cb1d1c01b0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"9e55879116702e2efd352cb1d1c01b0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.052780 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.052695 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e83d90178041f29950db3df0e41da00c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-83.ec2.internal\" (UID: \"e83d90178041f29950db3df0e41da00c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.057289 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.057272 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:20.152969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.152939 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e55879116702e2efd352cb1d1c01b0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"9e55879116702e2efd352cb1d1c01b0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.153058 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.152972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e55879116702e2efd352cb1d1c01b0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"9e55879116702e2efd352cb1d1c01b0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.153058 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.152989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e83d90178041f29950db3df0e41da00c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-83.ec2.internal\" (UID: \"e83d90178041f29950db3df0e41da00c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.153058 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.153055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e83d90178041f29950db3df0e41da00c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-83.ec2.internal\" (UID: \"e83d90178041f29950db3df0e41da00c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.153171 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.153055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e55879116702e2efd352cb1d1c01b0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"9e55879116702e2efd352cb1d1c01b0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.153171 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.153056 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e55879116702e2efd352cb1d1c01b0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"9e55879116702e2efd352cb1d1c01b0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.158058 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.158040 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:20.258451 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.258372 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:20.302577 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.302542 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.306398 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.306378 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 16 17:30:20.359320 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.359279 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:20.459894 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.459861 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:20.560384 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.560354 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:20.661082 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.661048 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:20.672219 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.672195 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 17:30:20.672410 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.672372 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 17:30:20.689022 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.688998 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:30:20.761640 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.761612 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:20.778819 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.778777 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:25:20 +0000 UTC" deadline="2028-01-19 19:42:48.204757362 +0000 UTC" Jul 16 17:30:20.778819 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.778812 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13250h12m27.425949858s" Jul 16 17:30:20.844810 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:20.844779 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83d90178041f29950db3df0e41da00c.slice/crio-f8ae59b72fb2a9f17ee4d0370fa76a3a6cb1588f82bcad016e52a898b267c9f6 WatchSource:0}: Error finding container f8ae59b72fb2a9f17ee4d0370fa76a3a6cb1588f82bcad016e52a898b267c9f6: Status 404 returned error can't find the container with id f8ae59b72fb2a9f17ee4d0370fa76a3a6cb1588f82bcad016e52a898b267c9f6 Jul 16 17:30:20.850423 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.850402 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:30:20.862351 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.862332 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:20.872097 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.872074 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:30:20.874033 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.873996 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" event={"ID":"e83d90178041f29950db3df0e41da00c","Type":"ContainerStarted","Data":"f8ae59b72fb2a9f17ee4d0370fa76a3a6cb1588f82bcad016e52a898b267c9f6"} Jul 16 17:30:20.895512 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.895480 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:30:20.896646 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.896629 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:30:20.896742 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.896667 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:30:20.896742 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:20.896681 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:30:20.962772 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:20.962741 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:21.063237 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.063208 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:21.163829 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.163755 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 16 17:30:21.171626 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.171605 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:30:21.250690 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.250653 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 16 17:30:21.265710 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.265574 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:30:21.266792 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.266676 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 16 17:30:21.277355 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.277331 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:30:21.287331 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:21.287305 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e55879116702e2efd352cb1d1c01b0a.slice/crio-c1938f10e3226fdfbe4874f03dbb4b5168721c1081519a9932e01142c288a947 WatchSource:0}: Error finding container c1938f10e3226fdfbe4874f03dbb4b5168721c1081519a9932e01142c288a947: Status 404 returned error can't find the container with id c1938f10e3226fdfbe4874f03dbb4b5168721c1081519a9932e01142c288a947 Jul 16 17:30:21.727598 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.727557 2565 apiserver.go:52] "Watching apiserver" Jul 16 17:30:21.737365 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.737340 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 17:30:21.737705 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.737679 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-xbth4","openshift-cluster-node-tuning-operator/tuned-7xt6c","openshift-dns/node-resolver-fbsff","openshift-image-registry/node-ca-cr9ts","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-kk57w","kube-system/global-pull-secret-syncer-dddd5","kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm","openshift-multus/multus-additional-cni-plugins-c94x7","openshift-multus/multus-hw446","openshift-multus/network-metrics-daemon-wtj6z","openshift-network-diagnostics/network-check-target-9mcrs","openshift-network-operator/iptables-alerter-4xqgw"] Jul 16 17:30:21.740720 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.740701 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:21.743020 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.742998 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.744092 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.744070 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 17:30:21.744241 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.744122 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 17:30:21.744893 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.744877 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-889hd\"" Jul 16 17:30:21.745344 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.745324 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.745587 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.745431 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:30:21.746369 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.746347 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lhjqs\"" Jul 16 17:30:21.746508 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.746487 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 17:30:21.747790 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.747769 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:21.747903 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.747878 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:21.748324 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.748302 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 17:30:21.748418 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.748404 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 17:30:21.748476 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.748438 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4x4mt\"" Jul 16 17:30:21.750127 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.750111 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:21.750189 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.750171 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:21.751016 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.751001 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 17:30:21.752607 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.752590 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.755157 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.755143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.756454 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.756438 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 17:30:21.757587 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.757567 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 17:30:21.757684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.757572 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 17:30:21.757684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.757607 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xxlbg\"" Jul 16 17:30:21.757684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.757609 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 17:30:21.757833 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.757766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.757963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.757915 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 17:30:21.758035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.758016 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 17:30:21.758088 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.758057 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 17:30:21.758745 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.758729 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 17:30:21.759042 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.759024 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 17:30:21.759242 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.759226 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lzfkx\"" Jul 16 17:30:21.760307 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.760291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hw446" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.760987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-systemd\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-run\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761102 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-sys\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt6sh\" (UniqueName: \"kubernetes.io/projected/ae18d2a8-cbb4-4632-9b9e-0135a3d51768-kube-api-access-wt6sh\") pod \"node-resolver-fbsff\" (UID: \"ae18d2a8-cbb4-4632-9b9e-0135a3d51768\") " pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-run-openvswitch\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761251 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-var-lib-kubelet\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-systemd-units\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-run-systemd\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761497 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-etc-openvswitch\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-node-log\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761577 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-sysctl-d\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-sysctl-conf\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-lib-modules\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.761669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-tmp\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-log-socket\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50e0d455-a823-47e2-881b-bc2fadf9857b-ovn-node-metrics-cert\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqsbr\" (UniqueName: \"kubernetes.io/projected/50e0d455-a823-47e2-881b-bc2fadf9857b-kube-api-access-vqsbr\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-kubernetes\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-tuned\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.761981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-kubelet-config\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-slash\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-run-ovn\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50e0d455-a823-47e2-881b-bc2fadf9857b-ovnkube-script-lib\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-host\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744t4\" (UniqueName: \"kubernetes.io/projected/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-kube-api-access-744t4\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae18d2a8-cbb4-4632-9b9e-0135a3d51768-tmp-dir\") pod \"node-resolver-fbsff\" (UID: \"ae18d2a8-cbb4-4632-9b9e-0135a3d51768\") " pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-var-lib-openvswitch\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-modprobe-d\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.762472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762433 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae18d2a8-cbb4-4632-9b9e-0135a3d51768-hosts-file\") pod \"node-resolver-fbsff\" (UID: \"ae18d2a8-cbb4-4632-9b9e-0135a3d51768\") " pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762493 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-dbus\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-kubelet\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-run-ovn-kubernetes\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762626 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-cni-netd\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50e0d455-a823-47e2-881b-bc2fadf9857b-ovnkube-config\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50e0d455-a823-47e2-881b-bc2fadf9857b-env-overrides\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0e7f5fe-bb08-43f7-8995-3767512c6544-agent-certs\") pod \"konnectivity-agent-xbth4\" (UID: \"f0e7f5fe-bb08-43f7-8995-3767512c6544\") " pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7f5fe-bb08-43f7-8995-3767512c6544-konnectivity-ca\") pod \"konnectivity-agent-xbth4\" (UID: \"f0e7f5fe-bb08-43f7-8995-3767512c6544\") " pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-run-netns\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762923 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-cni-bin\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.763250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.762957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-sysconfig\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.763779 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.763329 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 17:30:21.763779 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.763331 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 17:30:21.763779 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.763715 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 17:30:21.764386 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.763783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:21.764386 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.763849 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:21.764386 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.763907 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gq6qt\"" Jul 16 17:30:21.764386 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.763940 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 17:30:21.764386 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.764150 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bfd2l\"" Jul 16 17:30:21.764386 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.764319 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 17:30:21.764386 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.764336 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 17:30:21.766624 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.766602 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:21.769160 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.769144 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:21.769640 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.769623 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:30:21.769715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.769701 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 17:30:21.770184 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.770160 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bfnch\"" Jul 16 17:30:21.770296 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.770275 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 17:30:21.770354 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.770294 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 17:30:21.773493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.773475 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:30:21.773584 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.773484 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 17:30:21.773643 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.773616 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g9pcn\"" Jul 16 17:30:21.773781 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.773759 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 17:30:21.852471 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.852436 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 17:30:21.863408 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-var-lib-kubelet\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.863539 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h494w\" (UniqueName: \"kubernetes.io/projected/4a86433b-1109-40d7-914e-4ef577d6b404-kube-api-access-h494w\") pod \"iptables-alerter-4xqgw\" (UID: \"4a86433b-1109-40d7-914e-4ef577d6b404\") " pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:21.863539 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-sys-fs\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.863539 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-run-systemd\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.863539 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-node-log\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.863539 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-sysctl-d\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.863539 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-lib-modules\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-tmp\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-node-log\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863547 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-run-systemd\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-kubelet-config\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-cni-dir\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863679 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-log-socket\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-sysctl-d\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqsbr\" (UniqueName: \"kubernetes.io/projected/50e0d455-a823-47e2-881b-bc2fadf9857b-kube-api-access-vqsbr\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863725 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-lib-modules\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-os-release\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-kubelet-config\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863779 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-cnibin\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-log-socket\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.863830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863802 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-run-netns\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863826 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-run-multus-certs\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-slash\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50e0d455-a823-47e2-881b-bc2fadf9857b-ovnkube-script-lib\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-host\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-slash\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744t4\" (UniqueName: \"kubernetes.io/projected/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-kube-api-access-744t4\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.863962 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae18d2a8-cbb4-4632-9b9e-0135a3d51768-tmp-dir\") pod \"node-resolver-fbsff\" (UID: \"ae18d2a8-cbb4-4632-9b9e-0135a3d51768\") " pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-modprobe-d\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-system-cni-dir\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrmv\" (UniqueName: \"kubernetes.io/projected/b46a4bae-3afa-45d4-8f34-b30e97e381fa-kube-api-access-dbrmv\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864124 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5pp\" (UniqueName: \"kubernetes.io/projected/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-kube-api-access-wr5pp\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-socket-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9b4f\" (UniqueName: \"kubernetes.io/projected/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-kube-api-access-j9b4f\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-system-cni-dir\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864311 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-host\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.864644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864363 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae18d2a8-cbb4-4632-9b9e-0135a3d51768-tmp-dir\") pod \"node-resolver-fbsff\" (UID: \"ae18d2a8-cbb4-4632-9b9e-0135a3d51768\") " pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-modprobe-d\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-daemon-config\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-run-ovn-kubernetes\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50e0d455-a823-47e2-881b-bc2fadf9857b-ovnkube-config\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0e7f5fe-bb08-43f7-8995-3767512c6544-agent-certs\") pod \"konnectivity-agent-xbth4\" (UID: \"f0e7f5fe-bb08-43f7-8995-3767512c6544\") " pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864584 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-run-ovn-kubernetes\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7f5fe-bb08-43f7-8995-3767512c6544-konnectivity-ca\") pod \"konnectivity-agent-xbth4\" (UID: \"f0e7f5fe-bb08-43f7-8995-3767512c6544\") " pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.864734 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50e0d455-a823-47e2-881b-bc2fadf9857b-ovnkube-script-lib\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865280 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50e0d455-a823-47e2-881b-bc2fadf9857b-ovnkube-config\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-sysconfig\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865378 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-sys\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865408 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-run-k8s-cni-cncf-io\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865456 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7f5fe-bb08-43f7-8995-3767512c6544-konnectivity-ca\") pod \"konnectivity-agent-xbth4\" (UID: \"f0e7f5fe-bb08-43f7-8995-3767512c6544\") " pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-run-netns\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-sysconfig\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.865521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-cni-bin\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865513 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-sys\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b46a4bae-3afa-45d4-8f34-b30e97e381fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-run-netns\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fsn\" (UniqueName: \"kubernetes.io/projected/c4a102ac-4212-4e83-a108-e2992a5007ff-kube-api-access-t4fsn\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-run-openvswitch\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865601 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-cni-bin\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-os-release\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865652 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4a102ac-4212-4e83-a108-e2992a5007ff-cni-binary-copy\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d4dfa86-8906-4d95-9195-734bfcc49c84-serviceca\") pod \"node-ca-cr9ts\" (UID: \"5d4dfa86-8906-4d95-9195-734bfcc49c84\") " pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865645 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-run-openvswitch\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hzz\" (UniqueName: \"kubernetes.io/projected/5d4dfa86-8906-4d95-9195-734bfcc49c84-kube-api-access-27hzz\") pod \"node-ca-cr9ts\" (UID: \"5d4dfa86-8906-4d95-9195-734bfcc49c84\") " pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a86433b-1109-40d7-914e-4ef577d6b404-iptables-alerter-script\") pod \"iptables-alerter-4xqgw\" (UID: \"4a86433b-1109-40d7-914e-4ef577d6b404\") " pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a86433b-1109-40d7-914e-4ef577d6b404-host-slash\") pod \"iptables-alerter-4xqgw\" (UID: \"4a86433b-1109-40d7-914e-4ef577d6b404\") " pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-var-lib-kubelet\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-var-lib-cni-bin\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-var-lib-cni-multus\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.866313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865862 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-hostroot\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-var-lib-kubelet\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-systemd-units\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-systemd-units\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-etc-openvswitch\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.865989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-sysctl-conf\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866042 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-etc-openvswitch\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866107 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-registration-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50e0d455-a823-47e2-881b-bc2fadf9857b-ovn-node-metrics-cert\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-kubernetes\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866208 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-sysctl-conf\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-tuned\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866244 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-device-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-etc-kubernetes\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-run-ovn\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d4dfa86-8906-4d95-9195-734bfcc49c84-host\") pod \"node-ca-cr9ts\" (UID: \"5d4dfa86-8906-4d95-9195-734bfcc49c84\") " pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:21.866969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-var-lib-openvswitch\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae18d2a8-cbb4-4632-9b9e-0135a3d51768-hosts-file\") pod \"node-resolver-fbsff\" (UID: \"ae18d2a8-cbb4-4632-9b9e-0135a3d51768\") " pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-dbus\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-socket-dir-parent\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-kubelet\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866458 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-cni-netd\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50e0d455-a823-47e2-881b-bc2fadf9857b-env-overrides\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-var-lib-openvswitch\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt6sh\" (UniqueName: \"kubernetes.io/projected/ae18d2a8-cbb4-4632-9b9e-0135a3d51768-kube-api-access-wt6sh\") pod \"node-resolver-fbsff\" (UID: \"ae18d2a8-cbb4-4632-9b9e-0135a3d51768\") " pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866557 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae18d2a8-cbb4-4632-9b9e-0135a3d51768-hosts-file\") pod \"node-resolver-fbsff\" (UID: \"ae18d2a8-cbb4-4632-9b9e-0135a3d51768\") " pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866560 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-cnibin\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-systemd\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-kubelet\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-run\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.867774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866693 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-conf-dir\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b46a4bae-3afa-45d4-8f34-b30e97e381fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b46a4bae-3afa-45d4-8f34-b30e97e381fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-etc-selinux\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-run-ovn\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-kubernetes\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866977 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-systemd\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.866995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-run\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.867038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50e0d455-a823-47e2-881b-bc2fadf9857b-host-cni-netd\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.867061 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.867375 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-dbus\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.867382 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret podName:9c53a8a9-49b1-48d7-acd3-6aaf56b998ed nodeName:}" failed. No retries permitted until 2026-07-16 17:30:22.367355717 +0000 UTC m=+3.038825923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret") pod "global-pull-secret-syncer-dddd5" (UID: "9c53a8a9-49b1-48d7-acd3-6aaf56b998ed") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.867538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50e0d455-a823-47e2-881b-bc2fadf9857b-env-overrides\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.867927 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-tmp\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.868505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.868268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0e7f5fe-bb08-43f7-8995-3767512c6544-agent-certs\") pod \"konnectivity-agent-xbth4\" (UID: \"f0e7f5fe-bb08-43f7-8995-3767512c6544\") " pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:21.869347 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.869162 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50e0d455-a823-47e2-881b-bc2fadf9857b-ovn-node-metrics-cert\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.870305 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.870282 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-etc-tuned\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.876194 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.876125 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" event={"ID":"9e55879116702e2efd352cb1d1c01b0a","Type":"ContainerStarted","Data":"c1938f10e3226fdfbe4874f03dbb4b5168721c1081519a9932e01142c288a947"} Jul 16 17:30:21.878253 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.878227 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqsbr\" (UniqueName: \"kubernetes.io/projected/50e0d455-a823-47e2-881b-bc2fadf9857b-kube-api-access-vqsbr\") pod \"ovnkube-node-kk57w\" (UID: \"50e0d455-a823-47e2-881b-bc2fadf9857b\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:21.880912 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.880894 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:30:21.880912 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.880916 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:30:21.880912 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.880927 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhbt for pod openshift-network-diagnostics/network-check-target-9mcrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:21.881087 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.880986 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt podName:86638b4f-8f78-4423-ac50-0c3216994241 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:22.380967808 +0000 UTC m=+3.052438007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhbt" (UniqueName: "kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt") pod "network-check-target-9mcrs" (UID: "86638b4f-8f78-4423-ac50-0c3216994241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:21.882854 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.881856 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-744t4\" (UniqueName: \"kubernetes.io/projected/4fd3a957-4163-4eaa-93f3-1c81207b8ffa-kube-api-access-744t4\") pod \"tuned-7xt6c\" (UID: \"4fd3a957-4163-4eaa-93f3-1c81207b8ffa\") " pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:21.884029 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.884008 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt6sh\" (UniqueName: \"kubernetes.io/projected/ae18d2a8-cbb4-4632-9b9e-0135a3d51768-kube-api-access-wt6sh\") pod \"node-resolver-fbsff\" (UID: \"ae18d2a8-cbb4-4632-9b9e-0135a3d51768\") " pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:21.925228 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.925199 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqjkc" Jul 16 17:30:21.938070 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.938043 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:30:21.947421 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.947380 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqjkc" Jul 16 17:30:21.967773 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.967745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-registration-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.967944 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.967783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-device-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.967944 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.967806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-etc-kubernetes\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.967944 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.967890 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-etc-kubernetes\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.967944 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.967906 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-registration-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.967944 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.967908 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-device-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.967944 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.967940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d4dfa86-8906-4d95-9195-734bfcc49c84-host\") pod \"node-ca-cr9ts\" (UID: \"5d4dfa86-8906-4d95-9195-734bfcc49c84\") " pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:21.968178 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.967980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-socket-dir-parent\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968178 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-cnibin\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.968178 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-socket-dir-parent\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968178 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.967981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d4dfa86-8906-4d95-9195-734bfcc49c84-host\") pod \"node-ca-cr9ts\" (UID: \"5d4dfa86-8906-4d95-9195-734bfcc49c84\") " pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:21.968178 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.968178 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968068 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-cnibin\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.968178 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.968178 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968148 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-conf-dir\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968178 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b46a4bae-3afa-45d4-8f34-b30e97e381fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b46a4bae-3afa-45d4-8f34-b30e97e381fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968222 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-etc-selinux\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-conf-dir\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968294 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-var-lib-kubelet\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968324 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h494w\" (UniqueName: \"kubernetes.io/projected/4a86433b-1109-40d7-914e-4ef577d6b404-kube-api-access-h494w\") pod \"iptables-alerter-4xqgw\" (UID: \"4a86433b-1109-40d7-914e-4ef577d6b404\") " pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-sys-fs\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-etc-selinux\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-cni-dir\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-var-lib-kubelet\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-os-release\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-cnibin\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-sys-fs\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-run-netns\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-cni-dir\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.968657 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.968480 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968590 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-run-multus-certs\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968618 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-os-release\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-system-cni-dir\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:21.968645 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs podName:0d1fc925-7c4d-4e7a-9bb1-8607d8524be7 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:22.468625532 +0000 UTC m=+3.140095717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs") pod "network-metrics-daemon-wtj6z" (UID: "0d1fc925-7c4d-4e7a-9bb1-8607d8524be7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b46a4bae-3afa-45d4-8f34-b30e97e381fa-system-cni-dir\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968680 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-cnibin\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968690 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b46a4bae-3afa-45d4-8f34-b30e97e381fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-run-netns\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrmv\" (UniqueName: \"kubernetes.io/projected/b46a4bae-3afa-45d4-8f34-b30e97e381fa-kube-api-access-dbrmv\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-run-multus-certs\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5pp\" (UniqueName: \"kubernetes.io/projected/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-kube-api-access-wr5pp\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-socket-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968800 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b46a4bae-3afa-45d4-8f34-b30e97e381fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9b4f\" (UniqueName: \"kubernetes.io/projected/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-kube-api-access-j9b4f\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-system-cni-dir\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.969521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968863 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-daemon-config\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-run-k8s-cni-cncf-io\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968972 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-system-cni-dir\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.968981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-run-k8s-cni-cncf-io\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b46a4bae-3afa-45d4-8f34-b30e97e381fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fsn\" (UniqueName: \"kubernetes.io/projected/c4a102ac-4212-4e83-a108-e2992a5007ff-kube-api-access-t4fsn\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-os-release\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969095 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4a102ac-4212-4e83-a108-e2992a5007ff-cni-binary-copy\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d4dfa86-8906-4d95-9195-734bfcc49c84-serviceca\") pod \"node-ca-cr9ts\" (UID: \"5d4dfa86-8906-4d95-9195-734bfcc49c84\") " pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hzz\" (UniqueName: \"kubernetes.io/projected/5d4dfa86-8906-4d95-9195-734bfcc49c84-kube-api-access-27hzz\") pod \"node-ca-cr9ts\" (UID: \"5d4dfa86-8906-4d95-9195-734bfcc49c84\") " pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a86433b-1109-40d7-914e-4ef577d6b404-iptables-alerter-script\") pod \"iptables-alerter-4xqgw\" (UID: \"4a86433b-1109-40d7-914e-4ef577d6b404\") " pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-os-release\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a86433b-1109-40d7-914e-4ef577d6b404-host-slash\") pod \"iptables-alerter-4xqgw\" (UID: \"4a86433b-1109-40d7-914e-4ef577d6b404\") " pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-var-lib-cni-bin\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-var-lib-cni-multus\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-hostroot\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969385 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-hostroot\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969426 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-var-lib-cni-bin\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.970330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969466 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4a102ac-4212-4e83-a108-e2992a5007ff-host-var-lib-cni-multus\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.971343 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969502 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4a102ac-4212-4e83-a108-e2992a5007ff-multus-daemon-config\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.971343 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969574 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a86433b-1109-40d7-914e-4ef577d6b404-host-slash\") pod \"iptables-alerter-4xqgw\" (UID: \"4a86433b-1109-40d7-914e-4ef577d6b404\") " pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:21.971343 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d4dfa86-8906-4d95-9195-734bfcc49c84-serviceca\") pod \"node-ca-cr9ts\" (UID: \"5d4dfa86-8906-4d95-9195-734bfcc49c84\") " pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:21.971343 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969639 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4a102ac-4212-4e83-a108-e2992a5007ff-cni-binary-copy\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:21.971343 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-socket-dir\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:21.971343 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.969861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a86433b-1109-40d7-914e-4ef577d6b404-iptables-alerter-script\") pod \"iptables-alerter-4xqgw\" (UID: \"4a86433b-1109-40d7-914e-4ef577d6b404\") " pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:21.971343 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:21.970217 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b46a4bae-3afa-45d4-8f34-b30e97e381fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:22.001652 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.001578 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h494w\" (UniqueName: \"kubernetes.io/projected/4a86433b-1109-40d7-914e-4ef577d6b404-kube-api-access-h494w\") pod \"iptables-alerter-4xqgw\" (UID: \"4a86433b-1109-40d7-914e-4ef577d6b404\") " pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:22.001924 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.001880 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5pp\" (UniqueName: \"kubernetes.io/projected/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-kube-api-access-wr5pp\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:22.002039 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.001958 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hzz\" (UniqueName: \"kubernetes.io/projected/5d4dfa86-8906-4d95-9195-734bfcc49c84-kube-api-access-27hzz\") pod \"node-ca-cr9ts\" (UID: \"5d4dfa86-8906-4d95-9195-734bfcc49c84\") " pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:22.002587 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.002565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrmv\" (UniqueName: \"kubernetes.io/projected/b46a4bae-3afa-45d4-8f34-b30e97e381fa-kube-api-access-dbrmv\") pod \"multus-additional-cni-plugins-c94x7\" (UID: \"b46a4bae-3afa-45d4-8f34-b30e97e381fa\") " pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:22.002999 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.002976 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fsn\" (UniqueName: \"kubernetes.io/projected/c4a102ac-4212-4e83-a108-e2992a5007ff-kube-api-access-t4fsn\") pod \"multus-hw446\" (UID: \"c4a102ac-4212-4e83-a108-e2992a5007ff\") " pod="openshift-multus/multus-hw446" Jul 16 17:30:22.003263 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.003236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9b4f\" (UniqueName: \"kubernetes.io/projected/ffca4f0d-0579-47b7-95a1-3c3671ddf39e-kube-api-access-j9b4f\") pod \"aws-ebs-csi-driver-node-dprtm\" (UID: \"ffca4f0d-0579-47b7-95a1-3c3671ddf39e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:22.050694 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.050665 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:22.056411 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.056379 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" Jul 16 17:30:22.064134 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:22.064106 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e7f5fe_bb08_43f7_8995_3767512c6544.slice/crio-c7737eedc86c931581744046676b76839bfeb2ed82aded19c384f27d4cf5bcf7 WatchSource:0}: Error finding container c7737eedc86c931581744046676b76839bfeb2ed82aded19c384f27d4cf5bcf7: Status 404 returned error can't find the container with id c7737eedc86c931581744046676b76839bfeb2ed82aded19c384f27d4cf5bcf7 Jul 16 17:30:22.067303 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:22.067275 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd3a957_4163_4eaa_93f3_1c81207b8ffa.slice/crio-eb355eafb2bf0c2d2b63ded2a63f16f805a100892da435a65579cf11f49837b3 WatchSource:0}: Error finding container eb355eafb2bf0c2d2b63ded2a63f16f805a100892da435a65579cf11f49837b3: Status 404 returned error can't find the container with id eb355eafb2bf0c2d2b63ded2a63f16f805a100892da435a65579cf11f49837b3 Jul 16 17:30:22.068270 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.068247 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fbsff" Jul 16 17:30:22.073538 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.073515 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:22.079467 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.079449 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cr9ts" Jul 16 17:30:22.083144 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:22.083119 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae18d2a8_cbb4_4632_9b9e_0135a3d51768.slice/crio-3355844b55c53ca962df447c072dd527f3b4a7213f766213a5b402757b06b1e5 WatchSource:0}: Error finding container 3355844b55c53ca962df447c072dd527f3b4a7213f766213a5b402757b06b1e5: Status 404 returned error can't find the container with id 3355844b55c53ca962df447c072dd527f3b4a7213f766213a5b402757b06b1e5 Jul 16 17:30:22.084351 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.084328 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" Jul 16 17:30:22.087413 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:22.087358 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e0d455_a823_47e2_881b_bc2fadf9857b.slice/crio-814d0057881788b07b22670d863f31a2c4e157587dd3978b04d81ce50c5f8838 WatchSource:0}: Error finding container 814d0057881788b07b22670d863f31a2c4e157587dd3978b04d81ce50c5f8838: Status 404 returned error can't find the container with id 814d0057881788b07b22670d863f31a2c4e157587dd3978b04d81ce50c5f8838 Jul 16 17:30:22.089816 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.089798 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c94x7" Jul 16 17:30:22.095145 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.095122 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hw446" Jul 16 17:30:22.099575 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.099554 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xqgw" Jul 16 17:30:22.099802 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:22.099779 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffca4f0d_0579_47b7_95a1_3c3671ddf39e.slice/crio-5489bc32e662dd263b20f547c5f1425a0e474289a669de886dce36dc861aa5a7 WatchSource:0}: Error finding container 5489bc32e662dd263b20f547c5f1425a0e474289a669de886dce36dc861aa5a7: Status 404 returned error can't find the container with id 5489bc32e662dd263b20f547c5f1425a0e474289a669de886dce36dc861aa5a7 Jul 16 17:30:22.111720 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:22.111630 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a102ac_4212_4e83_a108_e2992a5007ff.slice/crio-6a5f389c977f76d176ea517894ae0f3701fa7866ac64c416960370a26e2c2894 WatchSource:0}: Error finding container 6a5f389c977f76d176ea517894ae0f3701fa7866ac64c416960370a26e2c2894: Status 404 returned error can't find the container with id 6a5f389c977f76d176ea517894ae0f3701fa7866ac64c416960370a26e2c2894 Jul 16 17:30:22.114201 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:22.114177 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a86433b_1109_40d7_914e_4ef577d6b404.slice/crio-f6536f9f8be1176dbdfc0d7879073837851745ee05c407a167526039c2a7001b WatchSource:0}: Error finding container f6536f9f8be1176dbdfc0d7879073837851745ee05c407a167526039c2a7001b: Status 404 returned error can't find the container with id f6536f9f8be1176dbdfc0d7879073837851745ee05c407a167526039c2a7001b Jul 16 17:30:22.372307 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.372271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:22.372485 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:22.372432 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:22.372540 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:22.372511 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret podName:9c53a8a9-49b1-48d7-acd3-6aaf56b998ed nodeName:}" failed. No retries permitted until 2026-07-16 17:30:23.372488906 +0000 UTC m=+4.043959091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret") pod "global-pull-secret-syncer-dddd5" (UID: "9c53a8a9-49b1-48d7-acd3-6aaf56b998ed") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:22.473448 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.473405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:22.473661 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.473464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:22.473661 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:22.473599 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:22.473661 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:22.473613 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:30:22.473661 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:22.473637 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:30:22.473661 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:22.473646 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhbt for pod openshift-network-diagnostics/network-check-target-9mcrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:22.473974 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:22.473676 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs podName:0d1fc925-7c4d-4e7a-9bb1-8607d8524be7 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:23.473659485 +0000 UTC m=+4.145129669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs") pod "network-metrics-daemon-wtj6z" (UID: "0d1fc925-7c4d-4e7a-9bb1-8607d8524be7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:22.473974 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:22.473695 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt podName:86638b4f-8f78-4423-ac50-0c3216994241 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:23.473684965 +0000 UTC m=+4.145155148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhbt" (UniqueName: "kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt") pod "network-check-target-9mcrs" (UID: "86638b4f-8f78-4423-ac50-0c3216994241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:22.883513 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.882557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fbsff" event={"ID":"ae18d2a8-cbb4-4632-9b9e-0135a3d51768","Type":"ContainerStarted","Data":"3355844b55c53ca962df447c072dd527f3b4a7213f766213a5b402757b06b1e5"} Jul 16 17:30:22.884442 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.884401 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" event={"ID":"4fd3a957-4163-4eaa-93f3-1c81207b8ffa","Type":"ContainerStarted","Data":"eb355eafb2bf0c2d2b63ded2a63f16f805a100892da435a65579cf11f49837b3"} Jul 16 17:30:22.887326 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.887284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xbth4" event={"ID":"f0e7f5fe-bb08-43f7-8995-3767512c6544","Type":"ContainerStarted","Data":"c7737eedc86c931581744046676b76839bfeb2ed82aded19c384f27d4cf5bcf7"} Jul 16 17:30:22.889408 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.889382 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xqgw" event={"ID":"4a86433b-1109-40d7-914e-4ef577d6b404","Type":"ContainerStarted","Data":"f6536f9f8be1176dbdfc0d7879073837851745ee05c407a167526039c2a7001b"} Jul 16 17:30:22.892839 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.892413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hw446" event={"ID":"c4a102ac-4212-4e83-a108-e2992a5007ff","Type":"ContainerStarted","Data":"6a5f389c977f76d176ea517894ae0f3701fa7866ac64c416960370a26e2c2894"} Jul 16 17:30:22.896946 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.895512 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c94x7" event={"ID":"b46a4bae-3afa-45d4-8f34-b30e97e381fa","Type":"ContainerStarted","Data":"02e3b3546bc56047355d68b3996359ce3dc319ead78a65490ced7f9ebaa53562"} Jul 16 17:30:22.900665 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.900542 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" event={"ID":"ffca4f0d-0579-47b7-95a1-3c3671ddf39e","Type":"ContainerStarted","Data":"5489bc32e662dd263b20f547c5f1425a0e474289a669de886dce36dc861aa5a7"} Jul 16 17:30:22.904454 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.904375 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cr9ts" event={"ID":"5d4dfa86-8906-4d95-9195-734bfcc49c84","Type":"ContainerStarted","Data":"a5f36666e90604988921512f83785a769a86dabfaa2654724ba992261cf4efee"} Jul 16 17:30:22.909240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.909139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" event={"ID":"50e0d455-a823-47e2-881b-bc2fadf9857b","Type":"ContainerStarted","Data":"814d0057881788b07b22670d863f31a2c4e157587dd3978b04d81ce50c5f8838"} Jul 16 17:30:22.952911 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.952829 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:25:22 +0000 UTC" deadline="2027-12-24 05:15:23.41141225 +0000 UTC" Jul 16 17:30:22.952911 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:22.952898 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12611h45m0.458520084s" Jul 16 17:30:23.383317 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.383220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:23.383473 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.383443 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:23.383532 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.383515 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret podName:9c53a8a9-49b1-48d7-acd3-6aaf56b998ed nodeName:}" failed. No retries permitted until 2026-07-16 17:30:25.383494263 +0000 UTC m=+6.054964448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret") pod "global-pull-secret-syncer-dddd5" (UID: "9c53a8a9-49b1-48d7-acd3-6aaf56b998ed") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:23.484005 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.483911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:23.484005 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.483988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:23.484244 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.484163 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:30:23.484244 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.484187 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:30:23.484244 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.484201 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhbt for pod openshift-network-diagnostics/network-check-target-9mcrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:23.484373 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.484262 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt podName:86638b4f-8f78-4423-ac50-0c3216994241 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:25.484243642 +0000 UTC m=+6.155713833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhbt" (UniqueName: "kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt") pod "network-check-target-9mcrs" (UID: "86638b4f-8f78-4423-ac50-0c3216994241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:23.484734 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.484693 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:23.484837 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.484758 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs podName:0d1fc925-7c4d-4e7a-9bb1-8607d8524be7 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:25.484739616 +0000 UTC m=+6.156209842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs") pod "network-metrics-daemon-wtj6z" (UID: "0d1fc925-7c4d-4e7a-9bb1-8607d8524be7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:23.874686 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.874650 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:23.874960 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.874787 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:23.875259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.875237 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:23.875387 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.875342 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:23.875534 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.875412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:23.875534 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:23.875501 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:23.936027 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.935987 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e55879116702e2efd352cb1d1c01b0a" containerID="1e6f87bc33e2a1f5afc115dd8dbd42f440c0fd23bf3b40688e600b23441e668b" exitCode=0 Jul 16 17:30:23.936541 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.936041 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" event={"ID":"9e55879116702e2efd352cb1d1c01b0a","Type":"ContainerDied","Data":"1e6f87bc33e2a1f5afc115dd8dbd42f440c0fd23bf3b40688e600b23441e668b"} Jul 16 17:30:23.954082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.953841 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:25:22 +0000 UTC" deadline="2028-04-15 10:57:45.582413636 +0000 UTC" Jul 16 17:30:23.954082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:23.953876 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15329h27m21.628543082s" Jul 16 17:30:24.952024 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:24.951308 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" event={"ID":"9e55879116702e2efd352cb1d1c01b0a","Type":"ContainerStarted","Data":"cd8bad1e35a802b02d4f51a6966d8f6d53909a95302c3792f1104cab5b6ae968"} Jul 16 17:30:25.401952 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:25.401427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:25.401952 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.401651 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:25.401952 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.401718 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret podName:9c53a8a9-49b1-48d7-acd3-6aaf56b998ed nodeName:}" failed. No retries permitted until 2026-07-16 17:30:29.401699324 +0000 UTC m=+10.073169517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret") pod "global-pull-secret-syncer-dddd5" (UID: "9c53a8a9-49b1-48d7-acd3-6aaf56b998ed") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:25.502597 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:25.502243 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:25.502597 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:25.502311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:25.502597 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.502409 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:25.502597 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.502479 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:30:25.502597 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.502489 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs podName:0d1fc925-7c4d-4e7a-9bb1-8607d8524be7 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:29.502467576 +0000 UTC m=+10.173937785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs") pod "network-metrics-daemon-wtj6z" (UID: "0d1fc925-7c4d-4e7a-9bb1-8607d8524be7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:25.502597 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.502499 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:30:25.502597 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.502516 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhbt for pod openshift-network-diagnostics/network-check-target-9mcrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:25.502597 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.502568 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt podName:86638b4f-8f78-4423-ac50-0c3216994241 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:29.502549523 +0000 UTC m=+10.174019710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhbt" (UniqueName: "kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt") pod "network-check-target-9mcrs" (UID: "86638b4f-8f78-4423-ac50-0c3216994241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:25.872018 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:25.871983 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:25.872197 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:25.871990 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:25.872197 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.872134 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:25.872335 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.872195 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:25.872335 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:25.872242 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:25.872335 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:25.872319 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:27.873742 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:27.873711 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:27.874248 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:27.873811 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:27.874248 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:27.874197 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:27.874385 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:27.874284 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:27.874385 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:27.874365 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:27.874485 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:27.874467 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:29.434466 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:29.434421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:29.434987 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.434658 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:29.434987 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.434735 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret podName:9c53a8a9-49b1-48d7-acd3-6aaf56b998ed nodeName:}" failed. No retries permitted until 2026-07-16 17:30:37.434714238 +0000 UTC m=+18.106184463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret") pod "global-pull-secret-syncer-dddd5" (UID: "9c53a8a9-49b1-48d7-acd3-6aaf56b998ed") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:29.535302 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:29.535253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:29.535481 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:29.535405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:29.535481 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.535468 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:30:29.535581 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.535497 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:30:29.535581 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.535512 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhbt for pod openshift-network-diagnostics/network-check-target-9mcrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:29.535581 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.535576 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt podName:86638b4f-8f78-4423-ac50-0c3216994241 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:37.535557255 +0000 UTC m=+18.207027446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhbt" (UniqueName: "kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt") pod "network-check-target-9mcrs" (UID: "86638b4f-8f78-4423-ac50-0c3216994241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:29.535740 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.535588 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:29.535740 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.535648 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs podName:0d1fc925-7c4d-4e7a-9bb1-8607d8524be7 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:37.535630441 +0000 UTC m=+18.207100654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs") pod "network-metrics-daemon-wtj6z" (UID: "0d1fc925-7c4d-4e7a-9bb1-8607d8524be7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:29.873276 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:29.872680 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:29.873276 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.872805 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:29.873276 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:29.872865 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:29.873276 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.872961 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:29.873276 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:29.873124 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:29.873276 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:29.873225 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:31.871338 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:31.871298 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:31.871876 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:31.871438 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:31.871876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:31.871793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:31.871876 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:31.871863 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:31.872085 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:31.871976 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:31.872085 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:31.872071 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:33.872421 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:33.872375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:33.872421 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:33.872397 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:33.872913 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:33.872490 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:33.872913 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:33.872499 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:33.872913 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:33.872588 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:33.872913 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:33.872644 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:35.871995 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:35.871959 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:35.871995 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:35.871969 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:35.872490 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:35.872161 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:35.872574 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:35.872551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:35.872687 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:35.872664 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:35.872775 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:35.872754 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:37.500459 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:37.500415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:37.500912 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.500584 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:37.500912 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.500649 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret podName:9c53a8a9-49b1-48d7-acd3-6aaf56b998ed nodeName:}" failed. No retries permitted until 2026-07-16 17:30:53.50063479 +0000 UTC m=+34.172104972 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret") pod "global-pull-secret-syncer-dddd5" (UID: "9c53a8a9-49b1-48d7-acd3-6aaf56b998ed") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:37.601106 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:37.601072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:37.601295 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:37.601133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:37.601295 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.601232 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:37.601295 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.601290 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:30:37.601432 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.601310 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:30:37.601432 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.601316 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs podName:0d1fc925-7c4d-4e7a-9bb1-8607d8524be7 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:53.601293003 +0000 UTC m=+34.272763184 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs") pod "network-metrics-daemon-wtj6z" (UID: "0d1fc925-7c4d-4e7a-9bb1-8607d8524be7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:37.601432 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.601321 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhbt for pod openshift-network-diagnostics/network-check-target-9mcrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:37.601432 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.601372 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt podName:86638b4f-8f78-4423-ac50-0c3216994241 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:53.601357759 +0000 UTC m=+34.272827959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhbt" (UniqueName: "kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt") pod "network-check-target-9mcrs" (UID: "86638b4f-8f78-4423-ac50-0c3216994241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:37.871325 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:37.871286 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:37.871325 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:37.871304 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:37.871532 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:37.871297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:37.871532 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.871389 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:37.871532 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.871458 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:37.871633 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:37.871533 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:39.872751 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:39.872718 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:39.873222 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:39.872798 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:39.873222 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:39.872824 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:39.873222 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:39.872891 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:39.873222 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:39.872943 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:39.873222 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:39.873024 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:41.871498 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:41.871459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:41.872100 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:41.871459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:41.872100 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:41.871604 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:41.872100 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:41.871459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:41.872100 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:41.871673 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:41.872100 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:41.871751 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:43.871901 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.871572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:43.872649 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.871572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:43.872649 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:43.872004 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:43.872649 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.871618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:43.872649 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:43.872081 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:43.872649 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:43.872122 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:43.992578 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.992493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" event={"ID":"ffca4f0d-0579-47b7-95a1-3c3671ddf39e","Type":"ContainerStarted","Data":"9557944c86a033a6d274aa1402398dc4f318a94e809215c80152641d15c554c3"} Jul 16 17:30:43.994223 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.994195 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cr9ts" event={"ID":"5d4dfa86-8906-4d95-9195-734bfcc49c84","Type":"ContainerStarted","Data":"ffccaea8d372e794b7d641c563d83f9b2389afefbf06dcf2fa166d064700c541"} Jul 16 17:30:43.997497 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.997468 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" event={"ID":"50e0d455-a823-47e2-881b-bc2fadf9857b","Type":"ContainerStarted","Data":"70f342ed62c6ca05a77296f005267a89dea1990c02506631fa273cdac8055967"} Jul 16 17:30:43.997624 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.997559 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" event={"ID":"50e0d455-a823-47e2-881b-bc2fadf9857b","Type":"ContainerStarted","Data":"b0176037e8f0899dab0383d67ec9818b382d76a636ea80ad96ea050e6b269883"} Jul 16 17:30:43.997624 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.997610 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" event={"ID":"50e0d455-a823-47e2-881b-bc2fadf9857b","Type":"ContainerStarted","Data":"6958bedc0094b9a7250000c2854b5801eca4f31d27f506b5e5ce460fa8e10b88"} Jul 16 17:30:43.997719 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.997657 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" event={"ID":"50e0d455-a823-47e2-881b-bc2fadf9857b","Type":"ContainerStarted","Data":"488085dee4fdc55a756b415e45654af09100437eca2e3004048fe4e4a03224ec"} Jul 16 17:30:43.997771 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.997726 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" event={"ID":"50e0d455-a823-47e2-881b-bc2fadf9857b","Type":"ContainerStarted","Data":"f724aa905b5d3250e059e0f4ecd951ea1003901677dfd180d4cb42dcc2986988"} Jul 16 17:30:43.997771 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.997765 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" event={"ID":"50e0d455-a823-47e2-881b-bc2fadf9857b","Type":"ContainerStarted","Data":"e340f4deac8044f3b7ed33c3df58e19e687c3962f3bc4303f2bc6c7307aafa1d"} Jul 16 17:30:43.999177 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:43.999148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" event={"ID":"e83d90178041f29950db3df0e41da00c","Type":"ContainerStarted","Data":"95248f5f82e8f65688c40876a709100068091dabbb31954805f912a90cea784a"} Jul 16 17:30:44.000704 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.000680 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fbsff" event={"ID":"ae18d2a8-cbb4-4632-9b9e-0135a3d51768","Type":"ContainerStarted","Data":"a37b3fee977b46be0304d0cf9218aff7ca8557e67d732e4055fc57ac884cbead"} Jul 16 17:30:44.002206 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.002178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" event={"ID":"4fd3a957-4163-4eaa-93f3-1c81207b8ffa","Type":"ContainerStarted","Data":"e52485f37be0ca8495e7838fdfe59e941f6f8f8e63043e6a2ed1170038a4c79c"} Jul 16 17:30:44.003760 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.003716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xbth4" event={"ID":"f0e7f5fe-bb08-43f7-8995-3767512c6544","Type":"ContainerStarted","Data":"e7cdfd9222694d119a0bc8da250ac7baaeba7a79ad7e4f313464fe869a4a82ec"} Jul 16 17:30:44.005247 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.005219 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hw446" event={"ID":"c4a102ac-4212-4e83-a108-e2992a5007ff","Type":"ContainerStarted","Data":"312f2187419d1f183fa6b9b08d500836588a841c767675f00d9e598dce437701"} Jul 16 17:30:44.009587 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.009565 2565 generic.go:356] "Generic (PLEG): container finished" podID="b46a4bae-3afa-45d4-8f34-b30e97e381fa" containerID="0e4a337ea88d1b8ab88f2301ac9b62f4d30561445f88de295bf988be57248450" exitCode=0 Jul 16 17:30:44.009686 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.009599 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c94x7" event={"ID":"b46a4bae-3afa-45d4-8f34-b30e97e381fa","Type":"ContainerDied","Data":"0e4a337ea88d1b8ab88f2301ac9b62f4d30561445f88de295bf988be57248450"} Jul 16 17:30:44.012459 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.012419 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cr9ts" podStartSLOduration=1.977860395 podStartE2EDuration="23.01240852s" totalImagesPullingTime="21.034548125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:44.012058872 +0000 UTC m=+24.683529077" watchObservedRunningTime="2026-07-16 17:30:44.01240852 +0000 UTC m=+24.683878724" Jul 16 17:30:44.012544 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.012503 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" podStartSLOduration=22.012495884 podStartE2EDuration="22.012495884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:24.969348566 +0000 UTC m=+5.640818774" watchObservedRunningTime="2026-07-16 17:30:44.012495884 +0000 UTC m=+24.683966090" Jul 16 17:30:44.032403 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.032364 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7xt6c" podStartSLOduration=2.9658588569999997 podStartE2EDuration="24.032347492s" totalImagesPullingTime="21.066488635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:44.032005764 +0000 UTC m=+24.703475960" watchObservedRunningTime="2026-07-16 17:30:44.032347492 +0000 UTC m=+24.703817695" Jul 16 17:30:44.057435 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.057383 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fbsff" podStartSLOduration=8.463336507 podStartE2EDuration="24.057367483s" totalImagesPullingTime="15.594030976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:44.056402857 +0000 UTC m=+24.727873062" watchObservedRunningTime="2026-07-16 17:30:44.057367483 +0000 UTC m=+24.728837686" Jul 16 17:30:44.076175 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.076121 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xbth4" podStartSLOduration=8.462659847 podStartE2EDuration="24.076096413s" totalImagesPullingTime="15.613436566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:44.074658026 +0000 UTC m=+24.746128232" watchObservedRunningTime="2026-07-16 17:30:44.076096413 +0000 UTC m=+24.747566674" Jul 16 17:30:44.179610 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.179544 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hw446" podStartSLOduration=2.158910067 podStartE2EDuration="23.179523505s" totalImagesPullingTime="21.020613438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:44.179090577 +0000 UTC m=+24.850560781" watchObservedRunningTime="2026-07-16 17:30:44.179523505 +0000 UTC m=+24.850993711" Jul 16 17:30:44.919003 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:44.918975 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 17:30:45.013936 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:45.013658 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xqgw" event={"ID":"4a86433b-1109-40d7-914e-4ef577d6b404","Type":"ContainerStarted","Data":"b939020864434ba278ee3f06c8a63284f2719735bf2969233ff051f77d5aa627"} Jul 16 17:30:45.015954 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:45.015901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" event={"ID":"ffca4f0d-0579-47b7-95a1-3c3671ddf39e","Type":"ContainerStarted","Data":"465a7786f4459306252e8b00a0ee2ec4f8bda3369dc182f50c5db2c500c1c90d"} Jul 16 17:30:45.043992 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:45.043940 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" podStartSLOduration=23.043919176 podStartE2EDuration="23.043919176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:44.198362739 +0000 UTC m=+24.869832944" watchObservedRunningTime="2026-07-16 17:30:45.043919176 +0000 UTC m=+25.715389381" Jul 16 17:30:45.825006 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:45.824881 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:30:44.918997384Z","UUID":"03429cb4-baf9-4b21-b14a-935e7f12874d","Handler":null,"Name":"","Endpoint":""} Jul 16 17:30:45.826804 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:45.826782 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 17:30:45.826923 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:45.826868 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 17:30:45.872113 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:45.872087 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:45.872231 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:45.872214 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:45.872623 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:45.872606 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:45.872724 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:45.872687 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:45.872887 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:45.872871 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:45.872977 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:45.872942 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:46.022349 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:46.022318 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" event={"ID":"50e0d455-a823-47e2-881b-bc2fadf9857b","Type":"ContainerStarted","Data":"896685279ce026fcb4afaf0b25bd29de95d8e1bff8aa67cbe0bd0b4443afff13"} Jul 16 17:30:46.892120 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:46.892077 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:46.892983 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:46.892959 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:46.911695 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:46.911644 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4xqgw" podStartSLOduration=4.893183126 podStartE2EDuration="25.911625725s" totalImagesPullingTime="21.018442599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:45.044075226 +0000 UTC m=+25.715545411" watchObservedRunningTime="2026-07-16 17:30:46.911625725 +0000 UTC m=+27.583095930" Jul 16 17:30:46.975168 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:46.975136 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:46.975591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:46.975573 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xbth4" Jul 16 17:30:47.027517 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:47.027485 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" event={"ID":"ffca4f0d-0579-47b7-95a1-3c3671ddf39e","Type":"ContainerStarted","Data":"0ebd09a39c842450d703b4da418c7c6e31987813fed4cc76115446b2270236bf"} Jul 16 17:30:47.072069 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:47.072021 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dprtm" podStartSLOduration=2.577715652 podStartE2EDuration="26.072005603s" totalImagesPullingTime="23.494289951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:47.071555532 +0000 UTC m=+27.743025736" watchObservedRunningTime="2026-07-16 17:30:47.072005603 +0000 UTC m=+27.743475801" Jul 16 17:30:47.871504 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:47.871308 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:47.871688 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:47.871594 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:47.871688 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:47.871439 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:47.871688 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:47.871683 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:47.871870 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:47.871308 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:47.871870 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:47.871740 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:49.035215 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.035028 2565 generic.go:356] "Generic (PLEG): container finished" podID="b46a4bae-3afa-45d4-8f34-b30e97e381fa" containerID="8e93e2c6a88991d8f1a77e271d69fd3dc0283a171051326e38fdad5bd80337b0" exitCode=0 Jul 16 17:30:49.036048 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.035112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c94x7" event={"ID":"b46a4bae-3afa-45d4-8f34-b30e97e381fa","Type":"ContainerDied","Data":"8e93e2c6a88991d8f1a77e271d69fd3dc0283a171051326e38fdad5bd80337b0"} Jul 16 17:30:49.039240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.039218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" event={"ID":"50e0d455-a823-47e2-881b-bc2fadf9857b","Type":"ContainerStarted","Data":"0406913e1884f0646ffd4b1ec4f9a16ebc201d9b8f654cf35333fc9b1fe20b98"} Jul 16 17:30:49.039496 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.039477 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:49.039596 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.039536 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:49.039596 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.039583 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:49.056044 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.056020 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:49.056162 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.056118 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:30:49.872870 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.872821 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:49.873054 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:49.872940 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:49.873054 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.872955 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:49.873054 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:49.872983 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:49.873146 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:49.873082 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:49.873204 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:49.873183 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:50.542521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:50.542425 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" podStartSLOduration=8.388548179 podStartE2EDuration="29.542405862s" totalImagesPullingTime="21.153857683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:49.134393983 +0000 UTC m=+29.805864186" watchObservedRunningTime="2026-07-16 17:30:50.542405862 +0000 UTC m=+31.213876066" Jul 16 17:30:50.543087 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:50.543069 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mcrs"] Jul 16 17:30:50.543182 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:50.543172 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:50.543262 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:50.543247 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:50.546050 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:50.546022 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dddd5"] Jul 16 17:30:50.546180 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:50.546132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:50.546237 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:50.546222 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:50.546700 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:50.546678 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtj6z"] Jul 16 17:30:50.546794 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:50.546783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:50.546905 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:50.546887 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:51.046192 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:51.046157 2565 generic.go:356] "Generic (PLEG): container finished" podID="b46a4bae-3afa-45d4-8f34-b30e97e381fa" containerID="9a04ffe6f99a00ac6b01125055bb645420b4d91ef208a261d8c883ea83e69e12" exitCode=0 Jul 16 17:30:51.046360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:51.046231 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c94x7" event={"ID":"b46a4bae-3afa-45d4-8f34-b30e97e381fa","Type":"ContainerDied","Data":"9a04ffe6f99a00ac6b01125055bb645420b4d91ef208a261d8c883ea83e69e12"} Jul 16 17:30:51.871214 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:51.871188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:51.871214 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:51.871216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:51.871665 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:51.871299 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:51.871665 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:51.871359 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:52.872215 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:52.872028 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:52.872539 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:52.872301 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:53.052025 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:53.051992 2565 generic.go:356] "Generic (PLEG): container finished" podID="b46a4bae-3afa-45d4-8f34-b30e97e381fa" containerID="12acd2a8e78feefcd95b4076d7b195248b398767feba36976a2c543e972e468d" exitCode=0 Jul 16 17:30:53.052152 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:53.052035 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c94x7" event={"ID":"b46a4bae-3afa-45d4-8f34-b30e97e381fa","Type":"ContainerDied","Data":"12acd2a8e78feefcd95b4076d7b195248b398767feba36976a2c543e972e468d"} Jul 16 17:30:53.523203 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:53.523099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:53.523335 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.523215 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:53.523335 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.523273 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret podName:9c53a8a9-49b1-48d7-acd3-6aaf56b998ed nodeName:}" failed. No retries permitted until 2026-07-16 17:31:25.523258668 +0000 UTC m=+66.194728849 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret") pod "global-pull-secret-syncer-dddd5" (UID: "9c53a8a9-49b1-48d7-acd3-6aaf56b998ed") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:30:53.623671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:53.623571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:53.623671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:53.623632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:53.623984 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.623735 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:53.623984 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.623757 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:30:53.623984 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.623775 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:30:53.623984 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.623786 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhbt for pod openshift-network-diagnostics/network-check-target-9mcrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:53.623984 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.623799 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs podName:0d1fc925-7c4d-4e7a-9bb1-8607d8524be7 nodeName:}" failed. No retries permitted until 2026-07-16 17:31:25.623785399 +0000 UTC m=+66.295255585 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs") pod "network-metrics-daemon-wtj6z" (UID: "0d1fc925-7c4d-4e7a-9bb1-8607d8524be7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:30:53.623984 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.623833 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt podName:86638b4f-8f78-4423-ac50-0c3216994241 nodeName:}" failed. No retries permitted until 2026-07-16 17:31:25.62381833 +0000 UTC m=+66.295288531 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhbt" (UniqueName: "kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt") pod "network-check-target-9mcrs" (UID: "86638b4f-8f78-4423-ac50-0c3216994241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:30:53.871482 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:53.871450 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:53.871681 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.871604 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:53.871681 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:53.871653 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:53.871770 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:53.871744 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:54.871646 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:54.871606 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:54.872110 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:54.871720 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dddd5" podUID="9c53a8a9-49b1-48d7-acd3-6aaf56b998ed" Jul 16 17:30:55.871750 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:55.871716 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:55.872260 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:55.871717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:55.872260 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:55.871862 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:30:55.872260 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:55.871935 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mcrs" podUID="86638b4f-8f78-4423-ac50-0c3216994241" Jul 16 17:30:56.143735 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.143703 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeReady" Jul 16 17:30:56.143921 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.143822 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 17:30:56.217636 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.217600 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mspz8"] Jul 16 17:30:56.224115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.224083 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.225327 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.225299 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fb9fq"] Jul 16 17:30:56.228920 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.228892 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 17:30:56.229033 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.228953 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nrlbj\"" Jul 16 17:30:56.229168 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.229152 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 17:30:56.229301 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.229277 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fb9fq" Jul 16 17:30:56.233082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.233052 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b9wlf\"" Jul 16 17:30:56.234279 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.234259 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 17:30:56.234375 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.234305 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 17:30:56.235095 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.235048 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mspz8"] Jul 16 17:30:56.236920 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.236902 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 17:30:56.243255 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.243233 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fb9fq"] Jul 16 17:30:56.341188 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.341144 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.341361 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.341205 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a9d4d9-4310-448d-be02-499a71b76856-tmp-dir\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.341361 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.341282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35a9d4d9-4310-448d-be02-499a71b76856-config-volume\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.341361 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.341339 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn7nz\" (UniqueName: \"kubernetes.io/projected/25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa-kube-api-access-fn7nz\") pod \"ingress-canary-fb9fq\" (UID: \"25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa\") " pod="openshift-ingress-canary/ingress-canary-fb9fq" Jul 16 17:30:56.341511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.341393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppjv\" (UniqueName: \"kubernetes.io/projected/35a9d4d9-4310-448d-be02-499a71b76856-kube-api-access-hppjv\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.341511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.341485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa-cert\") pod \"ingress-canary-fb9fq\" (UID: \"25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa\") " pod="openshift-ingress-canary/ingress-canary-fb9fq" Jul 16 17:30:56.442063 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.441982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa-cert\") pod \"ingress-canary-fb9fq\" (UID: \"25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa\") " pod="openshift-ingress-canary/ingress-canary-fb9fq" Jul 16 17:30:56.442063 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.442038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.442258 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:56.442132 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:30:56.442258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.442146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a9d4d9-4310-448d-be02-499a71b76856-tmp-dir\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.442258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.442192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35a9d4d9-4310-448d-be02-499a71b76856-config-volume\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.442258 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:56.442207 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls podName:35a9d4d9-4310-448d-be02-499a71b76856 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:56.94218736 +0000 UTC m=+37.613657565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls") pod "dns-default-mspz8" (UID: "35a9d4d9-4310-448d-be02-499a71b76856") : secret "dns-default-metrics-tls" not found Jul 16 17:30:56.442258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.442239 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn7nz\" (UniqueName: \"kubernetes.io/projected/25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa-kube-api-access-fn7nz\") pod \"ingress-canary-fb9fq\" (UID: \"25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa\") " pod="openshift-ingress-canary/ingress-canary-fb9fq" Jul 16 17:30:56.442478 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.442284 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppjv\" (UniqueName: \"kubernetes.io/projected/35a9d4d9-4310-448d-be02-499a71b76856-kube-api-access-hppjv\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.442528 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.442509 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a9d4d9-4310-448d-be02-499a71b76856-tmp-dir\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.442756 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.442733 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35a9d4d9-4310-448d-be02-499a71b76856-config-volume\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.446860 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.446811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa-cert\") pod \"ingress-canary-fb9fq\" (UID: \"25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa\") " pod="openshift-ingress-canary/ingress-canary-fb9fq" Jul 16 17:30:56.458798 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.458777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppjv\" (UniqueName: \"kubernetes.io/projected/35a9d4d9-4310-448d-be02-499a71b76856-kube-api-access-hppjv\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.459062 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.459042 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn7nz\" (UniqueName: \"kubernetes.io/projected/25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa-kube-api-access-fn7nz\") pod \"ingress-canary-fb9fq\" (UID: \"25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa\") " pod="openshift-ingress-canary/ingress-canary-fb9fq" Jul 16 17:30:56.543050 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.543010 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fb9fq" Jul 16 17:30:56.872193 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.872154 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:30:56.875526 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.875501 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:30:56.947121 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:56.947091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:56.947255 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:56.947202 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:30:56.947255 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:56.947254 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls podName:35a9d4d9-4310-448d-be02-499a71b76856 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:57.94724097 +0000 UTC m=+38.618711152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls") pod "dns-default-mspz8" (UID: "35a9d4d9-4310-448d-be02-499a71b76856") : secret "dns-default-metrics-tls" not found Jul 16 17:30:57.070604 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:57.070467 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fb9fq"] Jul 16 17:30:57.073416 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:30:57.073392 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c9bc80_1e04_45d6_8f5c_fe45c84ce3fa.slice/crio-42795a851af715c9cede48652e5ebb1dcefb4b4c45dd13c18ab7d7737d96ecbb WatchSource:0}: Error finding container 42795a851af715c9cede48652e5ebb1dcefb4b4c45dd13c18ab7d7737d96ecbb: Status 404 returned error can't find the container with id 42795a851af715c9cede48652e5ebb1dcefb4b4c45dd13c18ab7d7737d96ecbb Jul 16 17:30:57.871966 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:57.871907 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:30:57.872292 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:57.871910 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:30:57.875535 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:57.875513 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sbctx\"" Jul 16 17:30:57.876956 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:57.876937 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:30:57.876956 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:57.876965 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:30:57.877130 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:57.877040 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:30:57.877219 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:57.877206 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hn8wr\"" Jul 16 17:30:57.955531 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:57.955496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:57.955705 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:57.955676 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:30:57.955776 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:57.955755 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls podName:35a9d4d9-4310-448d-be02-499a71b76856 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:59.955733611 +0000 UTC m=+40.627203802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls") pod "dns-default-mspz8" (UID: "35a9d4d9-4310-448d-be02-499a71b76856") : secret "dns-default-metrics-tls" not found Jul 16 17:30:58.068241 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:58.068210 2565 generic.go:356] "Generic (PLEG): container finished" podID="b46a4bae-3afa-45d4-8f34-b30e97e381fa" containerID="a0e7b45f79de1eaffae8973770a600e8698dd77f163b01bf5e86d67a11291bad" exitCode=0 Jul 16 17:30:58.068429 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:58.068310 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c94x7" event={"ID":"b46a4bae-3afa-45d4-8f34-b30e97e381fa","Type":"ContainerDied","Data":"a0e7b45f79de1eaffae8973770a600e8698dd77f163b01bf5e86d67a11291bad"} Jul 16 17:30:58.069904 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:58.069879 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fb9fq" event={"ID":"25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa","Type":"ContainerStarted","Data":"42795a851af715c9cede48652e5ebb1dcefb4b4c45dd13c18ab7d7737d96ecbb"} Jul 16 17:30:59.076244 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:59.076208 2565 generic.go:356] "Generic (PLEG): container finished" podID="b46a4bae-3afa-45d4-8f34-b30e97e381fa" containerID="3a32be3f797d7e7b1af3ac48b252a205b964f749605f3f2a309a6b9c1fa4ba62" exitCode=0 Jul 16 17:30:59.076757 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:59.076253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c94x7" event={"ID":"b46a4bae-3afa-45d4-8f34-b30e97e381fa","Type":"ContainerDied","Data":"3a32be3f797d7e7b1af3ac48b252a205b964f749605f3f2a309a6b9c1fa4ba62"} Jul 16 17:30:59.972240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:30:59.972017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:30:59.972240 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:59.972180 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:30:59.972413 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:30:59.972250 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls podName:35a9d4d9-4310-448d-be02-499a71b76856 nodeName:}" failed. No retries permitted until 2026-07-16 17:31:03.972234368 +0000 UTC m=+44.643704549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls") pod "dns-default-mspz8" (UID: "35a9d4d9-4310-448d-be02-499a71b76856") : secret "dns-default-metrics-tls" not found Jul 16 17:31:00.079622 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:00.079587 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fb9fq" event={"ID":"25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa","Type":"ContainerStarted","Data":"ec5338765d9b92bbe5b12f649200ff1bcdae067e72baa739c1bc58da5db9b256"} Jul 16 17:31:00.082666 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:00.082639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c94x7" event={"ID":"b46a4bae-3afa-45d4-8f34-b30e97e381fa","Type":"ContainerStarted","Data":"82006b89df929921d19071c820abefb4f1d842c1ab55341d427c188ed4db49d3"} Jul 16 17:31:00.098575 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:00.098533 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fb9fq" podStartSLOduration=1.659457693 podStartE2EDuration="4.098518133s" totalImagesPullingTime="2.43906044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:31:00.097800668 +0000 UTC m=+40.769270873" watchObservedRunningTime="2026-07-16 17:31:00.098518133 +0000 UTC m=+40.769988337" Jul 16 17:31:00.125119 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:00.125078 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c94x7" podStartSLOduration=7.580090384 podStartE2EDuration="39.125061679s" totalImagesPullingTime="31.544971295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:31:00.123700833 +0000 UTC m=+40.795171037" watchObservedRunningTime="2026-07-16 17:31:00.125061679 +0000 UTC m=+40.796531883" Jul 16 17:31:03.999763 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:03.999724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:31:04.000256 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:31:03.999894 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:31:04.000256 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:31:03.999955 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls podName:35a9d4d9-4310-448d-be02-499a71b76856 nodeName:}" failed. No retries permitted until 2026-07-16 17:31:11.999940623 +0000 UTC m=+52.671410805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls") pod "dns-default-mspz8" (UID: "35a9d4d9-4310-448d-be02-499a71b76856") : secret "dns-default-metrics-tls" not found Jul 16 17:31:12.059598 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:12.059554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:31:12.060033 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:31:12.059700 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:31:12.060033 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:31:12.059756 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls podName:35a9d4d9-4310-448d-be02-499a71b76856 nodeName:}" failed. No retries permitted until 2026-07-16 17:31:28.059741984 +0000 UTC m=+68.731212165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls") pod "dns-default-mspz8" (UID: "35a9d4d9-4310-448d-be02-499a71b76856") : secret "dns-default-metrics-tls" not found Jul 16 17:31:21.064517 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:21.064487 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk57w" Jul 16 17:31:25.554699 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.554660 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:31:25.557703 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.557685 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:31:25.568903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.568880 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53a8a9-49b1-48d7-acd3-6aaf56b998ed-original-pull-secret\") pod \"global-pull-secret-syncer-dddd5\" (UID: \"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed\") " pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:31:25.655463 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.655427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:31:25.655636 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.655473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:31:25.658360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.658341 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:31:25.658550 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.658539 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:31:25.665796 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:31:25.665776 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 17:31:25.665911 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:31:25.665864 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs podName:0d1fc925-7c4d-4e7a-9bb1-8607d8524be7 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:29.66582521 +0000 UTC m=+130.337295408 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs") pod "network-metrics-daemon-wtj6z" (UID: "0d1fc925-7c4d-4e7a-9bb1-8607d8524be7") : secret "metrics-daemon-secret" not found Jul 16 17:31:25.668522 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.668505 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:31:25.679397 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.679374 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhbt\" (UniqueName: \"kubernetes.io/projected/86638b4f-8f78-4423-ac50-0c3216994241-kube-api-access-vhhbt\") pod \"network-check-target-9mcrs\" (UID: \"86638b4f-8f78-4423-ac50-0c3216994241\") " pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:31:25.682191 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.682170 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dddd5" Jul 16 17:31:25.792792 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.792765 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sbctx\"" Jul 16 17:31:25.801006 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.800980 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:31:25.805582 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.805435 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dddd5"] Jul 16 17:31:25.935182 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:25.935149 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mcrs"] Jul 16 17:31:25.939163 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:31:25.939133 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86638b4f_8f78_4423_ac50_0c3216994241.slice/crio-983b98afe6d0e2736d4d02d7ac25501d8889cdf8ad57e08c70d1272db673a318 WatchSource:0}: Error finding container 983b98afe6d0e2736d4d02d7ac25501d8889cdf8ad57e08c70d1272db673a318: Status 404 returned error can't find the container with id 983b98afe6d0e2736d4d02d7ac25501d8889cdf8ad57e08c70d1272db673a318 Jul 16 17:31:26.141961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:26.141871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mcrs" event={"ID":"86638b4f-8f78-4423-ac50-0c3216994241","Type":"ContainerStarted","Data":"983b98afe6d0e2736d4d02d7ac25501d8889cdf8ad57e08c70d1272db673a318"} Jul 16 17:31:26.142897 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:26.142829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dddd5" event={"ID":"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed","Type":"ContainerStarted","Data":"90d64d6b3b6b7a2f67b5bb2c9fc0381820c1ebd35072ee4ada8aac109fac6179"} Jul 16 17:31:28.070745 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:28.070705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:31:28.071259 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:31:28.070896 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:31:28.071259 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:31:28.070990 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls podName:35a9d4d9-4310-448d-be02-499a71b76856 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:00.070968004 +0000 UTC m=+100.742438189 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls") pod "dns-default-mspz8" (UID: "35a9d4d9-4310-448d-be02-499a71b76856") : secret "dns-default-metrics-tls" not found Jul 16 17:31:31.158717 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:31.158677 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mcrs" event={"ID":"86638b4f-8f78-4423-ac50-0c3216994241","Type":"ContainerStarted","Data":"51005f681ebbcf5963345324b0627ab4cde5b51fb4ef6bd7602428c8dc8da592"} Jul 16 17:31:31.159189 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:31.158778 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:31:31.160169 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:31.160146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dddd5" event={"ID":"9c53a8a9-49b1-48d7-acd3-6aaf56b998ed","Type":"ContainerStarted","Data":"5cdbaf499cedf589e5e23731f238722dbac46d48f8bb2447854543f7fc6555b6"} Jul 16 17:31:31.213265 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:31:31.213218 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9mcrs" podStartSLOduration=66.87101369 podStartE2EDuration="1m11.213202694s" totalImagesPullingTime="4.342189004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:31:31.189052086 +0000 UTC m=+71.860522305" watchObservedRunningTime="2026-07-16 17:31:31.213202694 +0000 UTC m=+71.884672875" Jul 16 17:32:00.073188 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:00.073036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:32:00.073759 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:00.073206 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:32:00.073759 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:00.073288 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls podName:35a9d4d9-4310-448d-be02-499a71b76856 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:04.073265274 +0000 UTC m=+164.744735456 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls") pod "dns-default-mspz8" (UID: "35a9d4d9-4310-448d-be02-499a71b76856") : secret "dns-default-metrics-tls" not found Jul 16 17:32:02.165265 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:02.165235 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9mcrs" Jul 16 17:32:02.188924 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:02.188867 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dddd5" podStartSLOduration=96.714539915 podStartE2EDuration="1m41.188831134s" totalImagesPullingTime="4.474291219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:31:31.213611795 +0000 UTC m=+71.885081999" watchObservedRunningTime="2026-07-16 17:32:02.188831134 +0000 UTC m=+102.860301338" Jul 16 17:32:14.902576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.902540 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw"] Jul 16 17:32:14.905298 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.905282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw" Jul 16 17:32:14.908144 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.908127 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:32:14.908144 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.908128 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 16 17:32:14.909145 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.909131 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hsrnb\"" Jul 16 17:32:14.917339 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.917318 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw"] Jul 16 17:32:14.974574 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.974542 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw"] Jul 16 17:32:14.977325 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.977311 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:14.979618 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.979594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbpn\" (UniqueName: \"kubernetes.io/projected/be313ab4-6436-466f-84db-20042d0ddd4a-kube-api-access-bkbpn\") pod \"volume-data-source-validator-78495597f6-gfvgw\" (UID: \"be313ab4-6436-466f-84db-20042d0ddd4a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw" Jul 16 17:32:14.980403 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.980387 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 16 17:32:14.980968 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.980955 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 16 17:32:14.982098 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.982077 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:32:14.982218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.982193 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cv6z9\"" Jul 16 17:32:14.982696 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.982676 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 16 17:32:14.991729 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:14.991703 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw"] Jul 16 17:32:15.080701 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.080673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/673405af-1260-4611-b0c9-f7ace3907095-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-mxfhw\" (UID: \"673405af-1260-4611-b0c9-f7ace3907095\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.080888 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.080718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/673405af-1260-4611-b0c9-f7ace3907095-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-mxfhw\" (UID: \"673405af-1260-4611-b0c9-f7ace3907095\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.080888 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.080749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vvb\" (UniqueName: \"kubernetes.io/projected/673405af-1260-4611-b0c9-f7ace3907095-kube-api-access-26vvb\") pod \"kube-storage-version-migrator-operator-799cb7f958-mxfhw\" (UID: \"673405af-1260-4611-b0c9-f7ace3907095\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.080888 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.080807 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbpn\" (UniqueName: \"kubernetes.io/projected/be313ab4-6436-466f-84db-20042d0ddd4a-kube-api-access-bkbpn\") pod \"volume-data-source-validator-78495597f6-gfvgw\" (UID: \"be313ab4-6436-466f-84db-20042d0ddd4a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw" Jul 16 17:32:15.090858 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.090804 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56"] Jul 16 17:32:15.093153 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.093131 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbpn\" (UniqueName: \"kubernetes.io/projected/be313ab4-6436-466f-84db-20042d0ddd4a-kube-api-access-bkbpn\") pod \"volume-data-source-validator-78495597f6-gfvgw\" (UID: \"be313ab4-6436-466f-84db-20042d0ddd4a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw" Jul 16 17:32:15.094030 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.093994 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-748b65cd7d-mqsgk"] Jul 16 17:32:15.094154 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.094131 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:15.097109 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.097087 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7st77\"" Jul 16 17:32:15.098589 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.097342 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 16 17:32:15.098744 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.097630 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 16 17:32:15.098744 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.097996 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:32:15.099586 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.099567 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-mbmr9"] Jul 16 17:32:15.099703 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.099686 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.102365 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.102348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.103858 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.103825 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 17:32:15.103937 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.103902 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 17:32:15.103980 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.103941 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 16 17:32:15.104875 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.104833 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jpg69\"" Jul 16 17:32:15.105007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.104990 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 16 17:32:15.105082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.105033 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 16 17:32:15.105082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.105069 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 16 17:32:15.105232 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.105155 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 17:32:15.105232 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.105215 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 16 17:32:15.105625 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.105603 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mrlpx\"" Jul 16 17:32:15.105722 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.105603 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 16 17:32:15.105722 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.105701 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 17:32:15.111005 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.110981 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56"] Jul 16 17:32:15.111822 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.111805 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 16 17:32:15.112054 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.112033 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-748b65cd7d-mqsgk"] Jul 16 17:32:15.116122 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.116052 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-mbmr9"] Jul 16 17:32:15.181583 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-serving-cert\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.181583 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-stats-auth\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.181583 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5sbf\" (UniqueName: \"kubernetes.io/projected/c21e59b9-8163-4bbc-bfc6-44abb123817f-kube-api-access-r5sbf\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.181583 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkwt\" (UniqueName: \"kubernetes.io/projected/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-kube-api-access-pjkwt\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.181860 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181598 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.181860 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/673405af-1260-4611-b0c9-f7ace3907095-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-mxfhw\" (UID: \"673405af-1260-4611-b0c9-f7ace3907095\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.181860 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181697 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-snapshots\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.181860 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181733 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.181860 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-tmp\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.181860 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.181860 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:15.182127 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/673405af-1260-4611-b0c9-f7ace3907095-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-mxfhw\" (UID: \"673405af-1260-4611-b0c9-f7ace3907095\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.182127 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181893 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.182127 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181911 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-default-certificate\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.182127 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vvb\" (UniqueName: \"kubernetes.io/projected/673405af-1260-4611-b0c9-f7ace3907095-kube-api-access-26vvb\") pod \"kube-storage-version-migrator-operator-799cb7f958-mxfhw\" (UID: \"673405af-1260-4611-b0c9-f7ace3907095\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.182127 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.181982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wcf\" (UniqueName: \"kubernetes.io/projected/6fb65ab5-9884-4bf2-b8e4-49882586debf-kube-api-access-99wcf\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:15.182280 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.182248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/673405af-1260-4611-b0c9-f7ace3907095-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-mxfhw\" (UID: \"673405af-1260-4611-b0c9-f7ace3907095\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.184188 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.184172 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/673405af-1260-4611-b0c9-f7ace3907095-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-mxfhw\" (UID: \"673405af-1260-4611-b0c9-f7ace3907095\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.190116 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.190095 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn"] Jul 16 17:32:15.193147 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.193130 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn" Jul 16 17:32:15.193872 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.193827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vvb\" (UniqueName: \"kubernetes.io/projected/673405af-1260-4611-b0c9-f7ace3907095-kube-api-access-26vvb\") pod \"kube-storage-version-migrator-operator-799cb7f958-mxfhw\" (UID: \"673405af-1260-4611-b0c9-f7ace3907095\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.196067 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.196051 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pwqsr\"" Jul 16 17:32:15.207335 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.207312 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn"] Jul 16 17:32:15.213465 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.213447 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw" Jul 16 17:32:15.283327 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.283463 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:15.283463 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.283421 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 17:32:15.283526 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.283488 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls podName:6fb65ab5-9884-4bf2-b8e4-49882586debf nodeName:}" failed. No retries permitted until 2026-07-16 17:32:15.78347283 +0000 UTC m=+116.454943012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls") pod "cluster-samples-operator-65db858894-dvg56" (UID: "6fb65ab5-9884-4bf2-b8e4-49882586debf") : secret "samples-operator-tls" not found Jul 16 17:32:15.283526 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283500 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.283526 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-default-certificate\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.283661 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wcf\" (UniqueName: \"kubernetes.io/projected/6fb65ab5-9884-4bf2-b8e4-49882586debf-kube-api-access-99wcf\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:15.283661 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mq6\" (UniqueName: \"kubernetes.io/projected/50244332-3ed3-4a66-93a7-e78a83546fe3-kube-api-access-f4mq6\") pod \"network-check-source-869dcfcdcd-5zxhn\" (UID: \"50244332-3ed3-4a66-93a7-e78a83546fe3\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn" Jul 16 17:32:15.283661 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-serving-cert\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.283661 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283625 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-stats-auth\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.283876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283650 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5sbf\" (UniqueName: \"kubernetes.io/projected/c21e59b9-8163-4bbc-bfc6-44abb123817f-kube-api-access-r5sbf\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.283876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkwt\" (UniqueName: \"kubernetes.io/projected/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-kube-api-access-pjkwt\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.283876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.283876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-snapshots\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.283876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.283876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.283834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-tmp\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.284158 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.283929 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:32:15.284158 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.283991 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:15.783973384 +0000 UTC m=+116.455443566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : secret "router-metrics-certs-default" not found Jul 16 17:32:15.284158 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.284048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.284308 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.284170 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-tmp\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.284308 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.284200 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:15.784183901 +0000 UTC m=+116.455654109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : configmap references non-existent config key: service-ca.crt Jul 16 17:32:15.284308 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.284235 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.284566 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.284548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-snapshots\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.285550 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.285533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" Jul 16 17:32:15.286791 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.286769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-default-certificate\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.286996 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.286977 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-serving-cert\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.287440 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.287421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-stats-auth\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.294784 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.294745 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkwt\" (UniqueName: \"kubernetes.io/projected/86d1a4e2-abcd-4c3f-947d-1cd87cca019d-kube-api-access-pjkwt\") pod \"insights-operator-68d9dd94cf-mbmr9\" (UID: \"86d1a4e2-abcd-4c3f-947d-1cd87cca019d\") " pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.295304 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.295272 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5sbf\" (UniqueName: \"kubernetes.io/projected/c21e59b9-8163-4bbc-bfc6-44abb123817f-kube-api-access-r5sbf\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.296088 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.296053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wcf\" (UniqueName: \"kubernetes.io/projected/6fb65ab5-9884-4bf2-b8e4-49882586debf-kube-api-access-99wcf\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:15.338370 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.338291 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw"] Jul 16 17:32:15.384489 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.384457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mq6\" (UniqueName: \"kubernetes.io/projected/50244332-3ed3-4a66-93a7-e78a83546fe3-kube-api-access-f4mq6\") pod \"network-check-source-869dcfcdcd-5zxhn\" (UID: \"50244332-3ed3-4a66-93a7-e78a83546fe3\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn" Jul 16 17:32:15.394284 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.394259 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mq6\" (UniqueName: \"kubernetes.io/projected/50244332-3ed3-4a66-93a7-e78a83546fe3-kube-api-access-f4mq6\") pod \"network-check-source-869dcfcdcd-5zxhn\" (UID: \"50244332-3ed3-4a66-93a7-e78a83546fe3\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn" Jul 16 17:32:15.417721 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.417694 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw"] Jul 16 17:32:15.420887 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:15.420860 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673405af_1260_4611_b0c9_f7ace3907095.slice/crio-e4023d49c043e3e3aad474f66959386f4116aab4e8263a6f1e7a196f03f224c2 WatchSource:0}: Error finding container e4023d49c043e3e3aad474f66959386f4116aab4e8263a6f1e7a196f03f224c2: Status 404 returned error can't find the container with id e4023d49c043e3e3aad474f66959386f4116aab4e8263a6f1e7a196f03f224c2 Jul 16 17:32:15.422634 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.422618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" Jul 16 17:32:15.505110 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.505080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn" Jul 16 17:32:15.558692 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.558357 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-mbmr9"] Jul 16 17:32:15.561954 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:15.561923 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d1a4e2_abcd_4c3f_947d_1cd87cca019d.slice/crio-5b79062feba40f779cfb6ebc510acb0f08b6b3bd7443b1b61ca6c016b98c8c06 WatchSource:0}: Error finding container 5b79062feba40f779cfb6ebc510acb0f08b6b3bd7443b1b61ca6c016b98c8c06: Status 404 returned error can't find the container with id 5b79062feba40f779cfb6ebc510acb0f08b6b3bd7443b1b61ca6c016b98c8c06 Jul 16 17:32:15.634668 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.634622 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn"] Jul 16 17:32:15.638159 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:15.638133 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50244332_3ed3_4a66_93a7_e78a83546fe3.slice/crio-44191342d75a74da81c284c253f9e731caa8270a8cfef35aae80579e2c024300 WatchSource:0}: Error finding container 44191342d75a74da81c284c253f9e731caa8270a8cfef35aae80579e2c024300: Status 404 returned error can't find the container with id 44191342d75a74da81c284c253f9e731caa8270a8cfef35aae80579e2c024300 Jul 16 17:32:15.788285 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.788250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.788438 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.788293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:15.788438 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:15.788344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:15.788438 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.788411 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:16.788394893 +0000 UTC m=+117.459865074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : configmap references non-existent config key: service-ca.crt Jul 16 17:32:15.788544 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.788460 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 17:32:15.788544 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.788507 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls podName:6fb65ab5-9884-4bf2-b8e4-49882586debf nodeName:}" failed. No retries permitted until 2026-07-16 17:32:16.788495439 +0000 UTC m=+117.459965621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls") pod "cluster-samples-operator-65db858894-dvg56" (UID: "6fb65ab5-9884-4bf2-b8e4-49882586debf") : secret "samples-operator-tls" not found Jul 16 17:32:15.788544 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.788460 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:32:15.788544 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:15.788540 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:16.788529994 +0000 UTC m=+117.460000183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : secret "router-metrics-certs-default" not found Jul 16 17:32:16.269859 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:16.269797 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" event={"ID":"86d1a4e2-abcd-4c3f-947d-1cd87cca019d","Type":"ContainerStarted","Data":"5b79062feba40f779cfb6ebc510acb0f08b6b3bd7443b1b61ca6c016b98c8c06"} Jul 16 17:32:16.271733 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:16.271694 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw" event={"ID":"be313ab4-6436-466f-84db-20042d0ddd4a","Type":"ContainerStarted","Data":"69ecba58e11f704e7c8970068db105e5db5952a9a835a8b2c7959df04f2535fd"} Jul 16 17:32:16.273677 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:16.273603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" event={"ID":"673405af-1260-4611-b0c9-f7ace3907095","Type":"ContainerStarted","Data":"e4023d49c043e3e3aad474f66959386f4116aab4e8263a6f1e7a196f03f224c2"} Jul 16 17:32:16.275626 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:16.275544 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn" event={"ID":"50244332-3ed3-4a66-93a7-e78a83546fe3","Type":"ContainerStarted","Data":"40c67768fc40ef26dfc95e2dc7af547e6c0921cab5080b0f2a7d982e48b97a17"} Jul 16 17:32:16.275626 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:16.275584 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn" event={"ID":"50244332-3ed3-4a66-93a7-e78a83546fe3","Type":"ContainerStarted","Data":"44191342d75a74da81c284c253f9e731caa8270a8cfef35aae80579e2c024300"} Jul 16 17:32:16.297212 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:16.296466 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-5zxhn" podStartSLOduration=1.296447553 podStartE2EDuration="1.296447553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:16.29577629 +0000 UTC m=+116.967246495" watchObservedRunningTime="2026-07-16 17:32:16.296447553 +0000 UTC m=+116.967917758" Jul 16 17:32:16.798507 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:16.798474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:16.798650 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:16.798517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:16.798650 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:16.798609 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 17:32:16.798724 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:16.798647 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:18.798625698 +0000 UTC m=+119.470095897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : configmap references non-existent config key: service-ca.crt Jul 16 17:32:16.798724 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:16.798708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:16.798805 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:16.798729 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls podName:6fb65ab5-9884-4bf2-b8e4-49882586debf nodeName:}" failed. No retries permitted until 2026-07-16 17:32:18.798704822 +0000 UTC m=+119.470175019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls") pod "cluster-samples-operator-65db858894-dvg56" (UID: "6fb65ab5-9884-4bf2-b8e4-49882586debf") : secret "samples-operator-tls" not found Jul 16 17:32:16.798805 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:16.798782 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:32:16.798935 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:16.798827 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:18.798814206 +0000 UTC m=+119.470284391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : secret "router-metrics-certs-default" not found Jul 16 17:32:17.280746 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:17.280661 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw" event={"ID":"be313ab4-6436-466f-84db-20042d0ddd4a","Type":"ContainerStarted","Data":"7f6509921a42b2ae80a212f8d3e3b85772e241e24ea0723647fecdb5ab88af1e"} Jul 16 17:32:17.299551 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:17.299497 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfvgw" podStartSLOduration=1.83872738 podStartE2EDuration="3.299481989s" totalImagesPullingTime="1.460754609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:17.298009702 +0000 UTC m=+117.969479906" watchObservedRunningTime="2026-07-16 17:32:17.299481989 +0000 UTC m=+117.970952193" Jul 16 17:32:18.286916 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.286819 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" event={"ID":"86d1a4e2-abcd-4c3f-947d-1cd87cca019d","Type":"ContainerStarted","Data":"85d98e2ced0ef3e1d6ab49c30b9ae4dc232346cb1a4cd8032187a17280268c61"} Jul 16 17:32:18.289120 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.289072 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" event={"ID":"673405af-1260-4611-b0c9-f7ace3907095","Type":"ContainerStarted","Data":"2839c57dcecb6aa6b68e97ee774cae0a562d6285862296adcfc8e52ace6bf851"} Jul 16 17:32:18.313121 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.313073 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" podStartSLOduration=0.689783151 podStartE2EDuration="3.313057906s" totalImagesPullingTime="2.623274755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:18.312808311 +0000 UTC m=+118.984278514" watchObservedRunningTime="2026-07-16 17:32:18.313057906 +0000 UTC m=+118.984528135" Jul 16 17:32:18.816469 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.816428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:18.816633 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.816531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:18.816633 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.816568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:18.816633 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:18.816584 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:32:18.816754 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:18.816651 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:22.816633957 +0000 UTC m=+123.488104138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : secret "router-metrics-certs-default" not found Jul 16 17:32:18.816754 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:18.816673 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:22.816659797 +0000 UTC m=+123.488129980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : configmap references non-existent config key: service-ca.crt Jul 16 17:32:18.816754 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:18.816685 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 17:32:18.816754 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:18.816735 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls podName:6fb65ab5-9884-4bf2-b8e4-49882586debf nodeName:}" failed. No retries permitted until 2026-07-16 17:32:22.816722052 +0000 UTC m=+123.488192241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls") pod "cluster-samples-operator-65db858894-dvg56" (UID: "6fb65ab5-9884-4bf2-b8e4-49882586debf") : secret "samples-operator-tls" not found Jul 16 17:32:18.880145 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.880086 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" podStartSLOduration=2.116880869 podStartE2EDuration="4.880070384s" totalImagesPullingTime="2.763189515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:18.340657189 +0000 UTC m=+119.012127393" watchObservedRunningTime="2026-07-16 17:32:18.880070384 +0000 UTC m=+119.551540587" Jul 16 17:32:18.880597 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.880581 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m7pfv"] Jul 16 17:32:18.883955 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.883930 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m7pfv"] Jul 16 17:32:18.884070 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.884048 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:18.907936 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.907718 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 17:32:18.907936 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.907726 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 17:32:18.909192 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:18.909016 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ppv6l\"" Jul 16 17:32:19.018556 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.018525 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb032e05-63e3-44ec-b314-e7839c25ad0f-data-volume\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.018556 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.018559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.018806 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.018647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2gtt\" (UniqueName: \"kubernetes.io/projected/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-api-access-r2gtt\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.018806 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.018680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.018806 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.018793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb032e05-63e3-44ec-b314-e7839c25ad0f-crio-socket\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.119469 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.119385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb032e05-63e3-44ec-b314-e7839c25ad0f-data-volume\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.119469 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.119421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.119643 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:19.119519 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:32:19.119643 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:19.119568 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:19.619554591 +0000 UTC m=+120.291024773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : secret "insights-runtime-extractor-tls" not found Jul 16 17:32:19.119643 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.119571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2gtt\" (UniqueName: \"kubernetes.io/projected/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-api-access-r2gtt\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.119643 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.119627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.119809 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.119682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb032e05-63e3-44ec-b314-e7839c25ad0f-crio-socket\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.119809 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.119776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb032e05-63e3-44ec-b314-e7839c25ad0f-data-volume\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.119809 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:19.119785 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:32:19.119959 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:19.119855 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:19.619825504 +0000 UTC m=+120.291295686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : configmap "kube-rbac-proxy" not found Jul 16 17:32:19.119959 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.119863 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb032e05-63e3-44ec-b314-e7839c25ad0f-crio-socket\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.135131 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.135101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2gtt\" (UniqueName: \"kubernetes.io/projected/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-api-access-r2gtt\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.623485 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.623444 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.623894 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:19.623566 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:32:19.623894 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:19.623564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:19.623894 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:19.623629 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:20.623611339 +0000 UTC m=+121.295081535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : configmap "kube-rbac-proxy" not found Jul 16 17:32:19.623894 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:19.623650 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:32:19.623894 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:19.623689 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:20.623678898 +0000 UTC m=+121.295149080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : secret "insights-runtime-extractor-tls" not found Jul 16 17:32:20.630351 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:20.630311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:20.630791 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:20.630422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:20.630791 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:20.630445 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:32:20.630791 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:20.630516 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:22.630496724 +0000 UTC m=+123.301966952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : configmap "kube-rbac-proxy" not found Jul 16 17:32:20.630791 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:20.630536 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:32:20.630791 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:20.630571 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:22.630560803 +0000 UTC m=+123.302030985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : secret "insights-runtime-extractor-tls" not found Jul 16 17:32:20.855375 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:20.855347 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fbsff_ae18d2a8-cbb4-4632-9b9e-0135a3d51768/dns-node-resolver/0.log" Jul 16 17:32:21.454997 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:21.454971 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cr9ts_5d4dfa86-8906-4d95-9195-734bfcc49c84/node-ca/0.log" Jul 16 17:32:22.650787 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:22.650751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:22.651207 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:22.650811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:22.651207 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:22.650939 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:32:22.651207 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:22.650950 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:32:22.651207 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:22.651042 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:26.650996461 +0000 UTC m=+127.322466642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : secret "insights-runtime-extractor-tls" not found Jul 16 17:32:22.651207 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:22.651061 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:26.651052963 +0000 UTC m=+127.322523146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : configmap "kube-rbac-proxy" not found Jul 16 17:32:22.656138 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:22.656112 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fb9fq_25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa/serve-healthcheck-canary/0.log" Jul 16 17:32:22.852457 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:22.852420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:22.852623 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:22.852470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:22.852623 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:22.852562 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:32:22.852623 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:22.852584 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:30.852568559 +0000 UTC m=+131.524038745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : configmap references non-existent config key: service-ca.crt Jul 16 17:32:22.852623 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:22.852609 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:30.852598614 +0000 UTC m=+131.524068796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : secret "router-metrics-certs-default" not found Jul 16 17:32:22.852760 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:22.852623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:22.852760 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:22.852699 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 17:32:22.852760 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:22.852722 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls podName:6fb65ab5-9884-4bf2-b8e4-49882586debf nodeName:}" failed. No retries permitted until 2026-07-16 17:32:30.852715069 +0000 UTC m=+131.524185251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls") pod "cluster-samples-operator-65db858894-dvg56" (UID: "6fb65ab5-9884-4bf2-b8e4-49882586debf") : secret "samples-operator-tls" not found Jul 16 17:32:23.649483 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.649447 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-744f8cfc9d-qdbx8"] Jul 16 17:32:23.653710 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.653695 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.660074 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.660055 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 17:32:23.660303 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.660288 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 17:32:23.660502 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.660489 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hbp2d\"" Jul 16 17:32:23.661164 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.661146 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 17:32:23.666252 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.666228 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 17:32:23.666925 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.666905 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-744f8cfc9d-qdbx8"] Jul 16 17:32:23.759603 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.759566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-trusted-ca\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.759603 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.759603 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-installation-pull-secrets\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.759801 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.759624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-image-registry-private-configuration\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.759801 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.759710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-ca-trust-extracted\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.759801 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.759753 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-bound-sa-token\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.759924 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.759825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.759924 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.759884 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74whg\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-kube-api-access-74whg\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.759990 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.759922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-certificates\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.861211 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74whg\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-kube-api-access-74whg\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.861211 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-certificates\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.861419 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-trusted-ca\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.861454 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-installation-pull-secrets\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.861486 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-image-registry-private-configuration\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.861561 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-ca-trust-extracted\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.861613 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-bound-sa-token\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.861662 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.861781 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:23.861760 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:32:23.861980 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:23.861956 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-744f8cfc9d-qdbx8: secret "image-registry-tls" not found Jul 16 17:32:23.862081 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861925 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-ca-trust-extracted\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.862081 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:23.862027 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls podName:c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:24.362007899 +0000 UTC m=+125.033478084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls") pod "image-registry-744f8cfc9d-qdbx8" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7") : secret "image-registry-tls" not found Jul 16 17:32:23.862081 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.861860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-certificates\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.862427 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.862408 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-trusted-ca\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.863889 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.863867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-installation-pull-secrets\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.863976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.863921 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-image-registry-private-configuration\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.876966 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.876942 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74whg\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-kube-api-access-74whg\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:23.878049 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:23.878033 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-bound-sa-token\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:24.365924 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:24.365884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:24.366105 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:24.366012 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:32:24.366105 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:24.366024 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-744f8cfc9d-qdbx8: secret "image-registry-tls" not found Jul 16 17:32:24.366105 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:24.366087 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls podName:c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:25.36606913 +0000 UTC m=+126.037539315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls") pod "image-registry-744f8cfc9d-qdbx8" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7") : secret "image-registry-tls" not found Jul 16 17:32:25.374151 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:25.374102 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:25.374559 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:25.374255 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:32:25.374559 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:25.374273 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-744f8cfc9d-qdbx8: secret "image-registry-tls" not found Jul 16 17:32:25.374559 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:25.374330 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls podName:c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:27.374314202 +0000 UTC m=+128.045784385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls") pod "image-registry-744f8cfc9d-qdbx8" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7") : secret "image-registry-tls" not found Jul 16 17:32:26.685030 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:26.684988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:26.685400 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:26.685082 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:26.685400 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:26.685148 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:32:26.685400 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:26.685200 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:32:26.685400 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:26.685215 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:34.685199364 +0000 UTC m=+135.356669550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : secret "insights-runtime-extractor-tls" not found Jul 16 17:32:26.685400 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:26.685254 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:34.685240374 +0000 UTC m=+135.356710556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : configmap "kube-rbac-proxy" not found Jul 16 17:32:27.390510 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:27.390479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:27.390674 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:27.390594 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:32:27.390674 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:27.390610 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-744f8cfc9d-qdbx8: secret "image-registry-tls" not found Jul 16 17:32:27.390674 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:27.390661 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls podName:c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7 nodeName:}" failed. No retries permitted until 2026-07-16 17:32:31.390647443 +0000 UTC m=+132.062117624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls") pod "image-registry-744f8cfc9d-qdbx8" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7") : secret "image-registry-tls" not found Jul 16 17:32:29.712434 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:29.712394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:32:29.712811 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:29.712554 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 17:32:29.712811 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:29.712619 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs podName:0d1fc925-7c4d-4e7a-9bb1-8607d8524be7 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:31.712604868 +0000 UTC m=+252.384075050 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs") pod "network-metrics-daemon-wtj6z" (UID: "0d1fc925-7c4d-4e7a-9bb1-8607d8524be7") : secret "metrics-daemon-secret" not found Jul 16 17:32:30.922814 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:30.922760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:30.922814 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:30.922831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:30.923330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:30.922885 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:30.923330 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:30.922978 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:46.922952574 +0000 UTC m=+147.594422758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : configmap references non-existent config key: service-ca.crt Jul 16 17:32:30.923330 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:30.923001 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:32:30.923330 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:30.923052 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs podName:c21e59b9-8163-4bbc-bfc6-44abb123817f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:46.923039308 +0000 UTC m=+147.594509490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs") pod "router-default-748b65cd7d-mqsgk" (UID: "c21e59b9-8163-4bbc-bfc6-44abb123817f") : secret "router-metrics-certs-default" not found Jul 16 17:32:30.925312 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:30.925285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fb65ab5-9884-4bf2-b8e4-49882586debf-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-dvg56\" (UID: \"6fb65ab5-9884-4bf2-b8e4-49882586debf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:31.013304 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:31.013276 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7st77\"" Jul 16 17:32:31.020876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:31.020848 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" Jul 16 17:32:31.150166 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:31.150140 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56"] Jul 16 17:32:31.329948 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:31.329912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" event={"ID":"6fb65ab5-9884-4bf2-b8e4-49882586debf","Type":"ContainerStarted","Data":"8f161631b28de0261d19b6906f241b21be76fcf7f6d7ca9c467000f442642de9"} Jul 16 17:32:31.427002 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:31.426967 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:31.429536 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:31.429476 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls\") pod \"image-registry-744f8cfc9d-qdbx8\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:31.462525 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:31.462501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:31.591578 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:31.591549 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-744f8cfc9d-qdbx8"] Jul 16 17:32:31.594126 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:31.594104 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c350cb_d4fe_4b3e_a0b0_4b4ec9fc04b7.slice/crio-8585c8dc2d7849be277025975f00329eebb5d40a7722a64ea13b5e33d0a79715 WatchSource:0}: Error finding container 8585c8dc2d7849be277025975f00329eebb5d40a7722a64ea13b5e33d0a79715: Status 404 returned error can't find the container with id 8585c8dc2d7849be277025975f00329eebb5d40a7722a64ea13b5e33d0a79715 Jul 16 17:32:32.333705 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:32.333672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" event={"ID":"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7","Type":"ContainerStarted","Data":"07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22"} Jul 16 17:32:32.334158 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:32.333724 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" event={"ID":"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7","Type":"ContainerStarted","Data":"8585c8dc2d7849be277025975f00329eebb5d40a7722a64ea13b5e33d0a79715"} Jul 16 17:32:32.334158 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:32.333758 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:32:32.358046 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:32.357987 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" podStartSLOduration=9.357949904 podStartE2EDuration="9.357949904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:32.35711137 +0000 UTC m=+133.028581575" watchObservedRunningTime="2026-07-16 17:32:32.357949904 +0000 UTC m=+133.029420109" Jul 16 17:32:33.338660 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:33.338573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" event={"ID":"6fb65ab5-9884-4bf2-b8e4-49882586debf","Type":"ContainerStarted","Data":"e36b3a12324ff0786fcca6cd01ace1ef8dfc8cbc62caa49e9db3c8601bcdf65b"} Jul 16 17:32:33.338660 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:33.338642 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" event={"ID":"6fb65ab5-9884-4bf2-b8e4-49882586debf","Type":"ContainerStarted","Data":"2d2cb5f620e5669a065605b50c55f34b63acb9900141d933a3f8a146386e1eff"} Jul 16 17:32:33.363188 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:33.363133 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-dvg56" podStartSLOduration=16.47163338 podStartE2EDuration="18.363115466s" totalImagesPullingTime="1.891482086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:33.362013052 +0000 UTC m=+134.033483257" watchObservedRunningTime="2026-07-16 17:32:33.363115466 +0000 UTC m=+134.034585669" Jul 16 17:32:34.754833 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:34.754786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:34.755233 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:34.754875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:34.755233 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:34.754948 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:32:34.755233 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:34.755014 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls podName:eb032e05-63e3-44ec-b314-e7839c25ad0f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:50.754999144 +0000 UTC m=+151.426469326 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m7pfv" (UID: "eb032e05-63e3-44ec-b314-e7839c25ad0f") : secret "insights-runtime-extractor-tls" not found Jul 16 17:32:34.755352 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:34.755336 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb032e05-63e3-44ec-b314-e7839c25ad0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:46.949857 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:46.949801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:46.950293 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:46.949922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:46.950478 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:46.950460 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21e59b9-8163-4bbc-bfc6-44abb123817f-service-ca-bundle\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:46.952196 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:46.952176 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c21e59b9-8163-4bbc-bfc6-44abb123817f-metrics-certs\") pod \"router-default-748b65cd7d-mqsgk\" (UID: \"c21e59b9-8163-4bbc-bfc6-44abb123817f\") " pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:47.221805 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:47.221731 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jpg69\"" Jul 16 17:32:47.229340 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:47.229318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:47.395939 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:47.395913 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-748b65cd7d-mqsgk"] Jul 16 17:32:47.397163 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:47.397139 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21e59b9_8163_4bbc_bfc6_44abb123817f.slice/crio-5bbc2da887be524d298ee38d5e9fbb9a2595afbd8db571ab878ec2bd9a03e625 WatchSource:0}: Error finding container 5bbc2da887be524d298ee38d5e9fbb9a2595afbd8db571ab878ec2bd9a03e625: Status 404 returned error can't find the container with id 5bbc2da887be524d298ee38d5e9fbb9a2595afbd8db571ab878ec2bd9a03e625 Jul 16 17:32:48.385710 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:48.385668 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-748b65cd7d-mqsgk" event={"ID":"c21e59b9-8163-4bbc-bfc6-44abb123817f","Type":"ContainerStarted","Data":"729e20a1b14e22196df93fe188f05238920e60caff9e4880d3d83c0cfb1be640"} Jul 16 17:32:48.386177 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:48.385728 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-748b65cd7d-mqsgk" event={"ID":"c21e59b9-8163-4bbc-bfc6-44abb123817f","Type":"ContainerStarted","Data":"5bbc2da887be524d298ee38d5e9fbb9a2595afbd8db571ab878ec2bd9a03e625"} Jul 16 17:32:48.411220 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:48.411167 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-748b65cd7d-mqsgk" podStartSLOduration=33.411151246 podStartE2EDuration="33.411151246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:48.410084385 +0000 UTC m=+149.081554588" watchObservedRunningTime="2026-07-16 17:32:48.411151246 +0000 UTC m=+149.082621456" Jul 16 17:32:49.230520 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.230481 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:49.232995 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.232972 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:49.388874 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.388828 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:49.390165 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.390146 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-748b65cd7d-mqsgk" Jul 16 17:32:49.835158 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.835127 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-744f8cfc9d-qdbx8"] Jul 16 17:32:49.840859 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.840817 2565 patch_prober.go:28] interesting pod/image-registry-744f8cfc9d-qdbx8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 16 17:32:49.840953 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.840889 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" podUID="c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:32:49.870040 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.870012 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-nj7vd"] Jul 16 17:32:49.872747 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.872728 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-nj7vd" Jul 16 17:32:49.875900 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.875878 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 17:32:49.875997 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.875951 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gkd56\"" Jul 16 17:32:49.875997 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.875964 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 17:32:49.885027 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.885009 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td"] Jul 16 17:32:49.886984 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.886969 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" Jul 16 17:32:49.890186 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.890161 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vk8ks\"" Jul 16 17:32:49.890675 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.890654 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 17:32:49.891751 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.891731 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-nj7vd"] Jul 16 17:32:49.908516 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.908491 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td"] Jul 16 17:32:49.972940 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:49.972909 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6fd55a34-6d28-4693-b9d4-aef3dc141941-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-nj4td\" (UID: \"6fd55a34-6d28-4693-b9d4-aef3dc141941\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" Jul 16 17:32:50.073355 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.073318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv789\" (UniqueName: \"kubernetes.io/projected/8c5557a6-144f-4be6-8e3f-1bab640da344-kube-api-access-mv789\") pod \"downloads-68c9cbb497-nj7vd\" (UID: \"8c5557a6-144f-4be6-8e3f-1bab640da344\") " pod="openshift-console/downloads-68c9cbb497-nj7vd" Jul 16 17:32:50.073549 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.073372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6fd55a34-6d28-4693-b9d4-aef3dc141941-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-nj4td\" (UID: \"6fd55a34-6d28-4693-b9d4-aef3dc141941\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" Jul 16 17:32:50.075817 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.075789 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6fd55a34-6d28-4693-b9d4-aef3dc141941-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-nj4td\" (UID: \"6fd55a34-6d28-4693-b9d4-aef3dc141941\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" Jul 16 17:32:50.174392 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.174305 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv789\" (UniqueName: \"kubernetes.io/projected/8c5557a6-144f-4be6-8e3f-1bab640da344-kube-api-access-mv789\") pod \"downloads-68c9cbb497-nj7vd\" (UID: \"8c5557a6-144f-4be6-8e3f-1bab640da344\") " pod="openshift-console/downloads-68c9cbb497-nj7vd" Jul 16 17:32:50.196493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.196469 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" Jul 16 17:32:50.208787 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.208757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv789\" (UniqueName: \"kubernetes.io/projected/8c5557a6-144f-4be6-8e3f-1bab640da344-kube-api-access-mv789\") pod \"downloads-68c9cbb497-nj7vd\" (UID: \"8c5557a6-144f-4be6-8e3f-1bab640da344\") " pod="openshift-console/downloads-68c9cbb497-nj7vd" Jul 16 17:32:50.324024 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.324002 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td"] Jul 16 17:32:50.326103 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:50.326076 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd55a34_6d28_4693_b9d4_aef3dc141941.slice/crio-fb8744dd572a341e78304d2dcd89397efbb1355fa779cc2f8dd097a274c47f0f WatchSource:0}: Error finding container fb8744dd572a341e78304d2dcd89397efbb1355fa779cc2f8dd097a274c47f0f: Status 404 returned error can't find the container with id fb8744dd572a341e78304d2dcd89397efbb1355fa779cc2f8dd097a274c47f0f Jul 16 17:32:50.396659 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.396625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" event={"ID":"6fd55a34-6d28-4693-b9d4-aef3dc141941","Type":"ContainerStarted","Data":"fb8744dd572a341e78304d2dcd89397efbb1355fa779cc2f8dd097a274c47f0f"} Jul 16 17:32:50.482204 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.482125 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-nj7vd" Jul 16 17:32:50.609529 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.609507 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-nj7vd"] Jul 16 17:32:50.611735 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:50.611705 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5557a6_144f_4be6_8e3f_1bab640da344.slice/crio-687ba142ff96a1f04944f9d70ecf217ae26156a7e9e1944215b30c1462e35d54 WatchSource:0}: Error finding container 687ba142ff96a1f04944f9d70ecf217ae26156a7e9e1944215b30c1462e35d54: Status 404 returned error can't find the container with id 687ba142ff96a1f04944f9d70ecf217ae26156a7e9e1944215b30c1462e35d54 Jul 16 17:32:50.778268 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.778189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:50.780509 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.780490 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb032e05-63e3-44ec-b314-e7839c25ad0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7pfv\" (UID: \"eb032e05-63e3-44ec-b314-e7839c25ad0f\") " pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:50.997352 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:50.997318 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ppv6l\"" Jul 16 17:32:51.004908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.004889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m7pfv" Jul 16 17:32:51.186669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.186647 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m7pfv"] Jul 16 17:32:51.401639 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.401557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-nj7vd" event={"ID":"8c5557a6-144f-4be6-8e3f-1bab640da344","Type":"ContainerStarted","Data":"687ba142ff96a1f04944f9d70ecf217ae26156a7e9e1944215b30c1462e35d54"} Jul 16 17:32:51.410601 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:51.410568 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb032e05_63e3_44ec_b314_e7839c25ad0f.slice/crio-fe523510f4a7aa921b5889c12f3a1256f0720269c07ce50b7f46dcdf60f43701 WatchSource:0}: Error finding container fe523510f4a7aa921b5889c12f3a1256f0720269c07ce50b7f46dcdf60f43701: Status 404 returned error can't find the container with id fe523510f4a7aa921b5889c12f3a1256f0720269c07ce50b7f46dcdf60f43701 Jul 16 17:32:51.927876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.927828 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4cdc6456-wdl4x"] Jul 16 17:32:51.931137 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.931116 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:51.939122 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.938996 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vhj5z\"" Jul 16 17:32:51.939246 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.939172 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 17:32:51.940192 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.940166 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 17:32:51.940192 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.940182 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 17:32:51.940624 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.940605 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 17:32:51.941282 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.941264 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 17:32:51.961026 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.960997 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4cdc6456-wdl4x"] Jul 16 17:32:51.987075 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.987049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-config\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:51.987216 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.987111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-oauth-serving-cert\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:51.987216 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.987136 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-oauth-config\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:51.987216 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.987185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-service-ca\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:51.987373 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.987222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjc7p\" (UniqueName: \"kubernetes.io/projected/c7bac37f-01ff-4f9d-9157-0eab98e3729f-kube-api-access-pjc7p\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:51.987373 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:51.987334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-serving-cert\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.087716 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.087680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-serving-cert\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.087908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.087728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-config\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.087908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.087778 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-oauth-serving-cert\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.087908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.087802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-oauth-config\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.087908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.087816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-service-ca\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.087908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.087850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjc7p\" (UniqueName: \"kubernetes.io/projected/c7bac37f-01ff-4f9d-9157-0eab98e3729f-kube-api-access-pjc7p\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.088666 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.088604 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-oauth-serving-cert\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.088831 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.088805 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-service-ca\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.089167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.089142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-config\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.090772 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.090747 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-serving-cert\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.090907 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.090770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-oauth-config\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.099908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.099885 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjc7p\" (UniqueName: \"kubernetes.io/projected/c7bac37f-01ff-4f9d-9157-0eab98e3729f-kube-api-access-pjc7p\") pod \"console-6c4cdc6456-wdl4x\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.242017 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.241938 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:32:52.403605 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.403459 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4cdc6456-wdl4x"] Jul 16 17:32:52.406617 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:52.406592 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bac37f_01ff_4f9d_9157_0eab98e3729f.slice/crio-b555e59a8bc0d9518635410f08d04c896951f5e518b775e2b12b373e5776da17 WatchSource:0}: Error finding container b555e59a8bc0d9518635410f08d04c896951f5e518b775e2b12b373e5776da17: Status 404 returned error can't find the container with id b555e59a8bc0d9518635410f08d04c896951f5e518b775e2b12b373e5776da17 Jul 16 17:32:52.408349 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.408298 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" event={"ID":"6fd55a34-6d28-4693-b9d4-aef3dc141941","Type":"ContainerStarted","Data":"01246744699b2f75cab1b1791394410aa70de9d4a071c9cf749f278f051dbacb"} Jul 16 17:32:52.408741 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.408651 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" Jul 16 17:32:52.410571 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.410461 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7pfv" event={"ID":"eb032e05-63e3-44ec-b314-e7839c25ad0f","Type":"ContainerStarted","Data":"fe523510f4a7aa921b5889c12f3a1256f0720269c07ce50b7f46dcdf60f43701"} Jul 16 17:32:52.414876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.414830 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" Jul 16 17:32:52.426701 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.426593 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-nj4td" podStartSLOduration=2.301064628 podStartE2EDuration="3.426574787s" totalImagesPullingTime="1.125510159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:52.426105058 +0000 UTC m=+153.097575262" watchObservedRunningTime="2026-07-16 17:32:52.426574787 +0000 UTC m=+153.098044992" Jul 16 17:32:52.848409 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.847964 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-rgxtq"] Jul 16 17:32:52.852106 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.851866 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:52.855245 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.855221 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 17:32:52.855460 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.855445 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 17:32:52.857593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.857568 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 17:32:52.857854 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.857824 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5xdzq\"" Jul 16 17:32:52.858166 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.858139 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 17:32:52.858364 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.858348 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 17:32:52.865014 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.864939 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-rgxtq"] Jul 16 17:32:52.894664 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.894613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:52.894857 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.894702 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skkmb\" (UniqueName: \"kubernetes.io/projected/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-kube-api-access-skkmb\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:52.894857 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.894830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:52.894985 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.894914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:52.996225 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.996184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:52.996421 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.996261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:52.996421 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.996319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:52.996421 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.996386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skkmb\" (UniqueName: \"kubernetes.io/projected/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-kube-api-access-skkmb\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:52.996600 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:52.996468 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 16 17:32:52.996600 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:52.996542 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-tls podName:440ed8b2-8190-4ba9-92bf-2cfc51c84e1f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:53.496522576 +0000 UTC m=+154.167992761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-tls") pod "prometheus-operator-64785b6675-rgxtq" (UID: "440ed8b2-8190-4ba9-92bf-2cfc51c84e1f") : secret "prometheus-operator-tls" not found Jul 16 17:32:52.997026 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:52.996998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:53.000642 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:53.000610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:53.009469 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:53.009425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skkmb\" (UniqueName: \"kubernetes.io/projected/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-kube-api-access-skkmb\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:53.416141 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:53.416100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4cdc6456-wdl4x" event={"ID":"c7bac37f-01ff-4f9d-9157-0eab98e3729f","Type":"ContainerStarted","Data":"b555e59a8bc0d9518635410f08d04c896951f5e518b775e2b12b373e5776da17"} Jul 16 17:32:53.502451 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:53.502411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:53.502651 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:53.502610 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 16 17:32:53.502718 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:53.502675 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-tls podName:440ed8b2-8190-4ba9-92bf-2cfc51c84e1f nodeName:}" failed. No retries permitted until 2026-07-16 17:32:54.502656452 +0000 UTC m=+155.174126638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-tls") pod "prometheus-operator-64785b6675-rgxtq" (UID: "440ed8b2-8190-4ba9-92bf-2cfc51c84e1f") : secret "prometheus-operator-tls" not found Jul 16 17:32:54.513502 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:54.513455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:54.525357 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:54.525113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/440ed8b2-8190-4ba9-92bf-2cfc51c84e1f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-rgxtq\" (UID: \"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:54.669572 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:54.669355 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" Jul 16 17:32:59.188054 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:59.187978 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-rgxtq"] Jul 16 17:32:59.191506 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:32:59.191463 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440ed8b2_8190_4ba9_92bf_2cfc51c84e1f.slice/crio-2099769f5f7042a12cd3cc047cc23946fddf39fa334da2e9dc44f7d06c7c9a8f WatchSource:0}: Error finding container 2099769f5f7042a12cd3cc047cc23946fddf39fa334da2e9dc44f7d06c7c9a8f: Status 404 returned error can't find the container with id 2099769f5f7042a12cd3cc047cc23946fddf39fa334da2e9dc44f7d06c7c9a8f Jul 16 17:32:59.236741 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:32:59.236698 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mspz8" podUID="35a9d4d9-4310-448d-be02-499a71b76856" Jul 16 17:32:59.441568 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:59.441525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" event={"ID":"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f","Type":"ContainerStarted","Data":"2099769f5f7042a12cd3cc047cc23946fddf39fa334da2e9dc44f7d06c7c9a8f"} Jul 16 17:32:59.443206 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:59.443172 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7pfv" event={"ID":"eb032e05-63e3-44ec-b314-e7839c25ad0f","Type":"ContainerStarted","Data":"e64a6ef9399130ca17cfd2b8935584434a75735ee14be095ff1599188bfc1551"} Jul 16 17:32:59.444798 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:59.444766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4cdc6456-wdl4x" event={"ID":"c7bac37f-01ff-4f9d-9157-0eab98e3729f","Type":"ContainerStarted","Data":"c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64"} Jul 16 17:32:59.444950 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:59.444782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mspz8" Jul 16 17:32:59.470888 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:59.470828 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c4cdc6456-wdl4x" podStartSLOduration=1.840488693 podStartE2EDuration="8.470807562s" totalImagesPullingTime="6.630318869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:32:59.469934606 +0000 UTC m=+160.141404807" watchObservedRunningTime="2026-07-16 17:32:59.470807562 +0000 UTC m=+160.142277764" Jul 16 17:32:59.841055 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:32:59.841026 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:33:00.884289 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:00.884240 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wtj6z" podUID="0d1fc925-7c4d-4e7a-9bb1-8607d8524be7" Jul 16 17:33:02.242994 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:02.242806 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:33:02.243583 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:02.243559 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:33:02.248864 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:02.248812 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:33:02.461498 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:02.461459 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:33:04.100251 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:04.100224 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:33:04.102765 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:04.102741 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a9d4d9-4310-448d-be02-499a71b76856-metrics-tls\") pod \"dns-default-mspz8\" (UID: \"35a9d4d9-4310-448d-be02-499a71b76856\") " pod="openshift-dns/dns-default-mspz8" Jul 16 17:33:04.248985 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:04.248952 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nrlbj\"" Jul 16 17:33:04.256587 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:04.256563 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mspz8" Jul 16 17:33:04.965210 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:04.965171 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4cdc6456-wdl4x"] Jul 16 17:33:11.108400 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.108365 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mspz8"] Jul 16 17:33:11.111192 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:33:11.111156 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a9d4d9_4310_448d_be02_499a71b76856.slice/crio-4991da731c0c9562d23e94660e9c30c8875c1f7ab8cf5be1ae0bb2aa1f9b9016 WatchSource:0}: Error finding container 4991da731c0c9562d23e94660e9c30c8875c1f7ab8cf5be1ae0bb2aa1f9b9016: Status 404 returned error can't find the container with id 4991da731c0c9562d23e94660e9c30c8875c1f7ab8cf5be1ae0bb2aa1f9b9016 Jul 16 17:33:11.492577 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.492533 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-nj7vd" event={"ID":"8c5557a6-144f-4be6-8e3f-1bab640da344","Type":"ContainerStarted","Data":"9f2382bf197bdf2d9c05fc30882af6fb344def4e625f86843eed26f006ce1ca7"} Jul 16 17:33:11.495069 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.492915 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-nj7vd" Jul 16 17:33:11.496184 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.496160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7pfv" event={"ID":"eb032e05-63e3-44ec-b314-e7839c25ad0f","Type":"ContainerStarted","Data":"fa6df5859c5eb3b3bf1e1896c21d340a1d3753b1e0c562f78fa2bd420e5a5680"} Jul 16 17:33:11.498471 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.498451 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" event={"ID":"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f","Type":"ContainerStarted","Data":"1303b8bfae87726a4355de20017a5cb9b88303b9095f9e4af4832c43e46e99a2"} Jul 16 17:33:11.498578 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.498495 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" event={"ID":"440ed8b2-8190-4ba9-92bf-2cfc51c84e1f","Type":"ContainerStarted","Data":"c8dfc600603f51165cf6d9b275096d0b545936de91bef65ae30191b4475b0c9f"} Jul 16 17:33:11.500340 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.500286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mspz8" event={"ID":"35a9d4d9-4310-448d-be02-499a71b76856","Type":"ContainerStarted","Data":"4991da731c0c9562d23e94660e9c30c8875c1f7ab8cf5be1ae0bb2aa1f9b9016"} Jul 16 17:33:11.511956 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.511881 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-nj7vd" Jul 16 17:33:11.533675 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.533602 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-nj7vd" podStartSLOduration=2.146066598 podStartE2EDuration="22.533583495s" totalImagesPullingTime="20.387516897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:11.531272811 +0000 UTC m=+172.202743018" watchObservedRunningTime="2026-07-16 17:33:11.533583495 +0000 UTC m=+172.205053698" Jul 16 17:33:11.581736 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:11.581674 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-rgxtq" podStartSLOduration=7.82252365 podStartE2EDuration="19.58164173s" totalImagesPullingTime="11.75911808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:11.580908776 +0000 UTC m=+172.252378981" watchObservedRunningTime="2026-07-16 17:33:11.58164173 +0000 UTC m=+172.253111935" Jul 16 17:33:13.281745 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.281711 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qwcvg"] Jul 16 17:33:13.309513 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.309479 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.312289 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.312260 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gbx8t\"" Jul 16 17:33:13.312562 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.312532 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 17:33:13.313280 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.312961 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 17:33:13.313280 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.313180 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 17:33:13.388575 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.388294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68f611a5-8e24-4dbe-a1fd-316365337579-metrics-client-ca\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.388575 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.388354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-textfile\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.388575 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.388392 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/68f611a5-8e24-4dbe-a1fd-316365337579-root\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.388575 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.388425 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-accelerators-collector-config\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.388575 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.388481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-tls\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.389160 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.388615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.389160 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.388698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68f611a5-8e24-4dbe-a1fd-316365337579-sys\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.389160 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.388720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-wtmp\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.389160 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.388751 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj57s\" (UniqueName: \"kubernetes.io/projected/68f611a5-8e24-4dbe-a1fd-316365337579-kube-api-access-lj57s\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.489878 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68f611a5-8e24-4dbe-a1fd-316365337579-sys\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.489878 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-wtmp\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.489878 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj57s\" (UniqueName: \"kubernetes.io/projected/68f611a5-8e24-4dbe-a1fd-316365337579-kube-api-access-lj57s\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.489878 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68f611a5-8e24-4dbe-a1fd-316365337579-metrics-client-ca\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.489878 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-textfile\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.489878 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/68f611a5-8e24-4dbe-a1fd-316365337579-root\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.489878 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-accelerators-collector-config\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.489878 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-tls\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.491671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.491671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/68f611a5-8e24-4dbe-a1fd-316365337579-root\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.491671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.489994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68f611a5-8e24-4dbe-a1fd-316365337579-sys\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.491671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.490111 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-wtmp\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.491671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.490190 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-textfile\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.491671 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:13.490281 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 17:33:13.491671 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:13.490349 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-tls podName:68f611a5-8e24-4dbe-a1fd-316365337579 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:13.990330967 +0000 UTC m=+174.661801152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-tls") pod "node-exporter-qwcvg" (UID: "68f611a5-8e24-4dbe-a1fd-316365337579") : secret "node-exporter-tls" not found Jul 16 17:33:13.491671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.491231 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68f611a5-8e24-4dbe-a1fd-316365337579-metrics-client-ca\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.491671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.491646 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-accelerators-collector-config\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.493560 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.493529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.511174 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.511144 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj57s\" (UniqueName: \"kubernetes.io/projected/68f611a5-8e24-4dbe-a1fd-316365337579-kube-api-access-lj57s\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.872179 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.871965 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:33:13.995290 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:13.995253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-tls\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:13.996108 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:13.995624 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 17:33:13.996108 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:13.995733 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-tls podName:68f611a5-8e24-4dbe-a1fd-316365337579 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:14.995711352 +0000 UTC m=+175.667181556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-tls") pod "node-exporter-qwcvg" (UID: "68f611a5-8e24-4dbe-a1fd-316365337579") : secret "node-exporter-tls" not found Jul 16 17:33:14.439401 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.439365 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:33:14.546123 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.546063 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:33:14.546496 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.546216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.558134 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.558083 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 17:33:14.558593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.558431 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 17:33:14.558593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.558525 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 17:33:14.558858 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.558740 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 17:33:14.558963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.558946 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 17:33:14.559140 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.559118 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 17:33:14.559352 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.559338 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 17:33:14.559537 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.559517 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zlcsb\"" Jul 16 17:33:14.559659 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.559550 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 17:33:14.559659 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.559648 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 17:33:14.702592 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702375 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-web-config\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702592 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702418 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702592 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5xdw\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-kube-api-access-x5xdw\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702592 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702935 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702599 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702935 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702648 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-config-out\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702935 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702935 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702935 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702866 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702935 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.702935 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.703295 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.703295 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.702982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804322 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804281 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804501 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-config-out\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804582 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804582 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804683 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804601 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804683 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804683 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804851 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804851 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804851 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-web-config\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804851 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.804851 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5xdw\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-kube-api-access-x5xdw\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.805100 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.804861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.806550 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.806307 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.808128 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.807825 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.809019 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.808974 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.810656 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.810301 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-config-out\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.810656 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.810428 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.810656 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.810598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.810874 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.810827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.811783 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.811754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.811990 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.811967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.812782 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.812759 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.813288 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.813252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.813288 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.813259 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-web-config\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.818342 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.818296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5xdw\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-kube-api-access-x5xdw\") pod \"alertmanager-main-0\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.860030 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.859992 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:33:14.864293 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:14.864220 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" podUID="c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" containerName="registry" containerID="cri-o://07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22" gracePeriod=30 Jul 16 17:33:15.007223 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:15.006971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-tls\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:15.009996 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:15.009934 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68f611a5-8e24-4dbe-a1fd-316365337579-node-exporter-tls\") pod \"node-exporter-qwcvg\" (UID: \"68f611a5-8e24-4dbe-a1fd-316365337579\") " pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:15.127864 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:15.127756 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qwcvg" Jul 16 17:33:16.181089 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.181066 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:33:16.199187 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:33:16.198458 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f611a5_8e24_4dbe_a1fd_316365337579.slice/crio-01b5bce8c6b8961e3f1e1fb3139ba82a46a78cc70ba5e58cf101a7d2128e59fd WatchSource:0}: Error finding container 01b5bce8c6b8961e3f1e1fb3139ba82a46a78cc70ba5e58cf101a7d2128e59fd: Status 404 returned error can't find the container with id 01b5bce8c6b8961e3f1e1fb3139ba82a46a78cc70ba5e58cf101a7d2128e59fd Jul 16 17:33:16.317579 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.317548 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-certificates\") pod \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " Jul 16 17:33:16.317724 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.317601 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls\") pod \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " Jul 16 17:33:16.317724 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.317629 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74whg\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-kube-api-access-74whg\") pod \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " Jul 16 17:33:16.317724 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.317693 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-image-registry-private-configuration\") pod \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " Jul 16 17:33:16.317917 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.317754 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-bound-sa-token\") pod \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " Jul 16 17:33:16.317917 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.317810 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-trusted-ca\") pod \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " Jul 16 17:33:16.317917 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.317863 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-ca-trust-extracted\") pod \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " Jul 16 17:33:16.317917 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.317907 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-installation-pull-secrets\") pod \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\" (UID: \"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7\") " Jul 16 17:33:16.319465 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.319433 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:16.321722 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.321659 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:16.325783 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.325638 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:33:16.328252 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.328210 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:16.328522 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.328491 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:16.329243 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.329202 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-kube-api-access-74whg" (OuterVolumeSpecName: "kube-api-access-74whg") pod "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7"). InnerVolumeSpecName "kube-api-access-74whg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:16.332776 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.332749 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:33:16.339592 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.339565 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" (UID: "c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:33:16.397985 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.397956 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:33:16.402601 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:33:16.402568 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4383218_5bac_44f6_bc83_387bbe73f29c.slice/crio-d73270282a591aa94097a2999f6576d7bc3970fd2f98209000fcd8413e9670c7 WatchSource:0}: Error finding container d73270282a591aa94097a2999f6576d7bc3970fd2f98209000fcd8413e9670c7: Status 404 returned error can't find the container with id d73270282a591aa94097a2999f6576d7bc3970fd2f98209000fcd8413e9670c7 Jul 16 17:33:16.418711 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.418683 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-bound-sa-token\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:16.419505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.419485 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-trusted-ca\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:16.420167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.419904 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-ca-trust-extracted\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:16.420167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.419934 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-installation-pull-secrets\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:16.420167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.419953 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-certificates\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:16.420167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.419969 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:16.420167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.419986 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74whg\" (UniqueName: \"kubernetes.io/projected/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-kube-api-access-74whg\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:16.420167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.420002 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7-image-registry-private-configuration\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:16.457381 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.457351 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-f5f78957b-zphtm"] Jul 16 17:33:16.457808 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.457790 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" containerName="registry" Jul 16 17:33:16.457808 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.457809 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" containerName="registry" Jul 16 17:33:16.457966 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.457899 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" containerName="registry" Jul 16 17:33:16.477922 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.477892 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.487961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.486202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f5f78957b-zphtm"] Jul 16 17:33:16.487961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.486373 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 16 17:33:16.487961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.486951 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 16 17:33:16.487961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.487158 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 16 17:33:16.487961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.487370 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e2pum43h155hn\"" Jul 16 17:33:16.487961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.487543 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 16 17:33:16.487961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.487716 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lrp58\"" Jul 16 17:33:16.488380 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.487973 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 16 17:33:16.526901 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.526865 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7pfv" event={"ID":"eb032e05-63e3-44ec-b314-e7839c25ad0f","Type":"ContainerStarted","Data":"66b4590cfb50d77981bdc5f3e3a4aa36cc48e5283df84654a788117037fb2d89"} Jul 16 17:33:16.530121 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.530086 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwcvg" event={"ID":"68f611a5-8e24-4dbe-a1fd-316365337579","Type":"ContainerStarted","Data":"01b5bce8c6b8961e3f1e1fb3139ba82a46a78cc70ba5e58cf101a7d2128e59fd"} Jul 16 17:33:16.532316 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.532289 2565 generic.go:356] "Generic (PLEG): container finished" podID="c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" containerID="07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22" exitCode=0 Jul 16 17:33:16.532437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.532338 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" Jul 16 17:33:16.532437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.532414 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" event={"ID":"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7","Type":"ContainerDied","Data":"07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22"} Jul 16 17:33:16.532549 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.532456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-744f8cfc9d-qdbx8" event={"ID":"c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7","Type":"ContainerDied","Data":"8585c8dc2d7849be277025975f00329eebb5d40a7722a64ea13b5e33d0a79715"} Jul 16 17:33:16.532549 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.532484 2565 scope.go:117] "RemoveContainer" containerID="07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22" Jul 16 17:33:16.535437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.535366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mspz8" event={"ID":"35a9d4d9-4310-448d-be02-499a71b76856","Type":"ContainerStarted","Data":"8d1b6f562ab1f52f3d959b588d2a0abfeca485dc44f8b81dfde425d3b67fe60e"} Jul 16 17:33:16.537261 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.537240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerStarted","Data":"d73270282a591aa94097a2999f6576d7bc3970fd2f98209000fcd8413e9670c7"} Jul 16 17:33:16.546582 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.546555 2565 scope.go:117] "RemoveContainer" containerID="07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22" Jul 16 17:33:16.546927 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:16.546866 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22\": container with ID starting with 07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22 not found: ID does not exist" containerID="07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22" Jul 16 17:33:16.547019 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.546932 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22"} err="failed to get container status \"07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22\": rpc error: code = NotFound desc = could not find container \"07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22\": container with ID starting with 07fdd6380eeb49ccc7c09b62b41d10b4023fd40f91a4a1796a115465da78dd22 not found: ID does not exist" Jul 16 17:33:16.568156 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.568102 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m7pfv" podStartSLOduration=33.911437832 podStartE2EDuration="58.56808708s" totalImagesPullingTime="24.656649248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:32:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:16.56690184 +0000 UTC m=+177.238372055" watchObservedRunningTime="2026-07-16 17:33:16.56808708 +0000 UTC m=+177.239557283" Jul 16 17:33:16.601396 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.601155 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-744f8cfc9d-qdbx8"] Jul 16 17:33:16.604923 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.604893 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-744f8cfc9d-qdbx8"] Jul 16 17:33:16.621305 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.621263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.621305 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.621306 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d48n2\" (UniqueName: \"kubernetes.io/projected/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-kube-api-access-d48n2\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.621530 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.621368 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.621530 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.621469 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-metrics-client-ca\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.621530 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.621519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-grpc-tls\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.621689 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.621560 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.621689 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.621631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.621788 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.621720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-tls\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.722815 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.722778 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.723010 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.722886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-tls\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.723010 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.722922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.723010 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.722953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d48n2\" (UniqueName: \"kubernetes.io/projected/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-kube-api-access-d48n2\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.723181 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.723036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.723181 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.723091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-metrics-client-ca\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.723181 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.723126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-grpc-tls\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.723181 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.723164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.724033 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.724007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-metrics-client-ca\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.726520 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.726453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.726520 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.726453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.727138 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.727111 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.727471 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.727450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.727797 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.727773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-grpc-tls\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.730065 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.730038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-secret-thanos-querier-tls\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.740482 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.740436 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d48n2\" (UniqueName: \"kubernetes.io/projected/f3e255e5-d6e8-4fce-a1eb-1db19da83c80-kube-api-access-d48n2\") pod \"thanos-querier-f5f78957b-zphtm\" (UID: \"f3e255e5-d6e8-4fce-a1eb-1db19da83c80\") " pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:16.796186 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:16.796093 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:17.125374 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.125341 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f5f78957b-zphtm"] Jul 16 17:33:17.492607 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:33:17.492529 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e255e5_d6e8_4fce_a1eb_1db19da83c80.slice/crio-2cdf7347fd2492eb7e259dcb4adc0808caf6565e3e3c70f71c257b5514f764fc WatchSource:0}: Error finding container 2cdf7347fd2492eb7e259dcb4adc0808caf6565e3e3c70f71c257b5514f764fc: Status 404 returned error can't find the container with id 2cdf7347fd2492eb7e259dcb4adc0808caf6565e3e3c70f71c257b5514f764fc Jul 16 17:33:17.546545 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.546504 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mspz8" event={"ID":"35a9d4d9-4310-448d-be02-499a71b76856","Type":"ContainerStarted","Data":"ce60ea7bd0f362b533de19c35133d50f7539dc4a679a91932321037c8c364ffa"} Jul 16 17:33:17.547053 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.546810 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mspz8" Jul 16 17:33:17.548612 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.548534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" event={"ID":"f3e255e5-d6e8-4fce-a1eb-1db19da83c80","Type":"ContainerStarted","Data":"2cdf7347fd2492eb7e259dcb4adc0808caf6565e3e3c70f71c257b5514f764fc"} Jul 16 17:33:17.877946 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.877909 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7" path="/var/lib/kubelet/pods/c2c350cb-d4fe-4b3e-a0b0-4b4ec9fc04b7/volumes" Jul 16 17:33:17.897297 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.897242 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mspz8" podStartSLOduration=136.815345014 podStartE2EDuration="2m21.897220997s" totalImagesPullingTime="5.081875983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:17.585947572 +0000 UTC m=+178.257417778" watchObservedRunningTime="2026-07-16 17:33:17.897220997 +0000 UTC m=+178.568691200" Jul 16 17:33:17.898180 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.898154 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-597f4687df-q7z9v"] Jul 16 17:33:17.926767 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.926740 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:17.941223 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.941200 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 17:33:17.941507 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.941490 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 17:33:17.942423 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.942403 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1ldtvbrsaqt0q\"" Jul 16 17:33:17.942423 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.942418 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cpvgr\"" Jul 16 17:33:17.942576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.942557 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 17:33:17.942880 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.942862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 17:33:17.955499 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:17.955477 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-597f4687df-q7z9v"] Jul 16 17:33:18.039466 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.039428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5bnq\" (UniqueName: \"kubernetes.io/projected/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-kube-api-access-f5bnq\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.039623 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.039529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-audit-log\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.039623 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.039586 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-metrics-server-audit-profiles\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.039748 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.039713 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.039801 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.039759 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-client-ca-bundle\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.039872 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.039817 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-secret-metrics-server-client-certs\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.039963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.039938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-secret-metrics-server-tls\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.141198 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.141117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-secret-metrics-server-tls\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.141198 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.141180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5bnq\" (UniqueName: \"kubernetes.io/projected/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-kube-api-access-f5bnq\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.141412 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.141300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-audit-log\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.141412 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.141340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-metrics-server-audit-profiles\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.141412 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.141369 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.141412 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.141395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-client-ca-bundle\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.141621 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.141445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-secret-metrics-server-client-certs\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.142183 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.142099 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-audit-log\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.142599 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.142545 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.142721 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.142635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-metrics-server-audit-profiles\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.144522 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.144495 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-secret-metrics-server-client-certs\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.144621 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.144550 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-secret-metrics-server-tls\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.144713 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.144692 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-client-ca-bundle\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.179320 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.179288 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5bnq\" (UniqueName: \"kubernetes.io/projected/c3ad62a1-601f-4bb1-acc5-c567cc32f9cb-kube-api-access-f5bnq\") pod \"metrics-server-597f4687df-q7z9v\" (UID: \"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb\") " pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.238731 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.238705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:18.277276 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.277245 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f79d86fbb-hkrhg"] Jul 16 17:33:18.307113 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.307084 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.309425 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.309358 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f79d86fbb-hkrhg"] Jul 16 17:33:18.329236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.327083 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x"] Jul 16 17:33:18.356055 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.355363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:18.363065 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.362758 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 17:33:18.363501 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.363481 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 16 17:33:18.364161 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.364134 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rw2dk\"" Jul 16 17:33:18.411781 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.411757 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x"] Jul 16 17:33:18.443715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.443208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-config\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.443715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.443266 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-serving-cert\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.443715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.443301 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-oauth-serving-cert\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.443715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.443330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qtr\" (UniqueName: \"kubernetes.io/projected/b1fafa62-69d1-4086-b75d-7b59e92ec22a-kube-api-access-n2qtr\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.443715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.443356 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14753c04-960a-41c7-8111-95101d4dfeba-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-b9n2x\" (UID: \"14753c04-960a-41c7-8111-95101d4dfeba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:18.443715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.443466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-oauth-config\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.443715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.443504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-trusted-ca-bundle\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.443715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.443527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4n8\" (UniqueName: \"kubernetes.io/projected/14753c04-960a-41c7-8111-95101d4dfeba-kube-api-access-zf4n8\") pod \"monitoring-plugin-5f68df7f58-b9n2x\" (UID: \"14753c04-960a-41c7-8111-95101d4dfeba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:18.443715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.443578 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-service-ca\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.512126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.512052 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-597f4687df-q7z9v"] Jul 16 17:33:18.523387 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:33:18.523362 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ad62a1_601f_4bb1_acc5_c567cc32f9cb.slice/crio-47693d71208bccf5534917ab1c1abfe3548a68b07ca670a1cedb5bd848a3769f WatchSource:0}: Error finding container 47693d71208bccf5534917ab1c1abfe3548a68b07ca670a1cedb5bd848a3769f: Status 404 returned error can't find the container with id 47693d71208bccf5534917ab1c1abfe3548a68b07ca670a1cedb5bd848a3769f Jul 16 17:33:18.544801 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.544768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-config\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.544927 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.544871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-serving-cert\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.544927 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.544903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-oauth-serving-cert\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.545193 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.544930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qtr\" (UniqueName: \"kubernetes.io/projected/b1fafa62-69d1-4086-b75d-7b59e92ec22a-kube-api-access-n2qtr\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.545193 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.544957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14753c04-960a-41c7-8111-95101d4dfeba-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-b9n2x\" (UID: \"14753c04-960a-41c7-8111-95101d4dfeba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:18.545193 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.545063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-oauth-config\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.545193 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.545106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-trusted-ca-bundle\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.545193 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.545133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4n8\" (UniqueName: \"kubernetes.io/projected/14753c04-960a-41c7-8111-95101d4dfeba-kube-api-access-zf4n8\") pod \"monitoring-plugin-5f68df7f58-b9n2x\" (UID: \"14753c04-960a-41c7-8111-95101d4dfeba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:18.545193 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.545183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-service-ca\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.545964 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.545940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-service-ca\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.546627 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.546603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-config\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.547059 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:18.547039 2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 16 17:33:18.547150 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:18.547104 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14753c04-960a-41c7-8111-95101d4dfeba-monitoring-plugin-cert podName:14753c04-960a-41c7-8111-95101d4dfeba nodeName:}" failed. No retries permitted until 2026-07-16 17:33:19.047085975 +0000 UTC m=+179.718556171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/14753c04-960a-41c7-8111-95101d4dfeba-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-b9n2x" (UID: "14753c04-960a-41c7-8111-95101d4dfeba") : secret "monitoring-plugin-cert" not found Jul 16 17:33:18.549288 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.549258 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-oauth-serving-cert\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.549670 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.549621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-trusted-ca-bundle\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.550088 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.550063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-serving-cert\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.552103 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.552059 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-oauth-config\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.555516 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.555494 2565 generic.go:356] "Generic (PLEG): container finished" podID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerID="81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b" exitCode=0 Jul 16 17:33:18.555614 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.555578 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerDied","Data":"81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b"} Jul 16 17:33:18.559073 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.559046 2565 generic.go:356] "Generic (PLEG): container finished" podID="68f611a5-8e24-4dbe-a1fd-316365337579" containerID="b16d535902dab7cbd6710a8de81e8cc86e3894f4718ba521bfa29afee1303b32" exitCode=0 Jul 16 17:33:18.559181 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.559121 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwcvg" event={"ID":"68f611a5-8e24-4dbe-a1fd-316365337579","Type":"ContainerDied","Data":"b16d535902dab7cbd6710a8de81e8cc86e3894f4718ba521bfa29afee1303b32"} Jul 16 17:33:18.562945 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.562916 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" event={"ID":"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb","Type":"ContainerStarted","Data":"47693d71208bccf5534917ab1c1abfe3548a68b07ca670a1cedb5bd848a3769f"} Jul 16 17:33:18.592454 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.592415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4n8\" (UniqueName: \"kubernetes.io/projected/14753c04-960a-41c7-8111-95101d4dfeba-kube-api-access-zf4n8\") pod \"monitoring-plugin-5f68df7f58-b9n2x\" (UID: \"14753c04-960a-41c7-8111-95101d4dfeba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:18.597099 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.597078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qtr\" (UniqueName: \"kubernetes.io/projected/b1fafa62-69d1-4086-b75d-7b59e92ec22a-kube-api-access-n2qtr\") pod \"console-7f79d86fbb-hkrhg\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.623078 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.622513 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:18.755546 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.755480 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-64fccd8557-gwgwd"] Jul 16 17:33:18.772159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.772132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.779074 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.777961 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 17:33:18.779074 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.778499 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 17:33:18.794318 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.785227 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 17:33:18.794318 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.786085 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 17:33:18.794318 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.786725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lm4kr\"" Jul 16 17:33:18.797066 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.797040 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64fccd8557-gwgwd"] Jul 16 17:33:18.802016 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.801971 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 17:33:18.802355 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.802335 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 17:33:18.849104 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.849070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06664ca-c52a-4f51-aa10-5ea66c04df7f-metrics-client-ca\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.849253 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.849133 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06664ca-c52a-4f51-aa10-5ea66c04df7f-serving-certs-ca-bundle\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.849253 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.849165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06664ca-c52a-4f51-aa10-5ea66c04df7f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.849253 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.849229 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-federate-client-tls\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.849437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.849267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76kd\" (UniqueName: \"kubernetes.io/projected/c06664ca-c52a-4f51-aa10-5ea66c04df7f-kube-api-access-x76kd\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.849437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.849342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-secret-telemeter-client\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.849437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.849381 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-telemeter-client-tls\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.849437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.849421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.860382 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.859214 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f79d86fbb-hkrhg"] Jul 16 17:33:18.863411 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:33:18.863335 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fafa62_69d1_4086_b75d_7b59e92ec22a.slice/crio-17e5113d8d3e46eb436f3ca344e5458706f6fb115af901b9d902617600e81fee WatchSource:0}: Error finding container 17e5113d8d3e46eb436f3ca344e5458706f6fb115af901b9d902617600e81fee: Status 404 returned error can't find the container with id 17e5113d8d3e46eb436f3ca344e5458706f6fb115af901b9d902617600e81fee Jul 16 17:33:18.953115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.950902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-federate-client-tls\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.953115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.950957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76kd\" (UniqueName: \"kubernetes.io/projected/c06664ca-c52a-4f51-aa10-5ea66c04df7f-kube-api-access-x76kd\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.953115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.951033 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-secret-telemeter-client\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.953115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.951072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-telemeter-client-tls\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.953115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.951112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.953115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.951148 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06664ca-c52a-4f51-aa10-5ea66c04df7f-metrics-client-ca\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.953115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.951192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06664ca-c52a-4f51-aa10-5ea66c04df7f-serving-certs-ca-bundle\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.953115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.951222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06664ca-c52a-4f51-aa10-5ea66c04df7f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.953115 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.953037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06664ca-c52a-4f51-aa10-5ea66c04df7f-metrics-client-ca\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.954150 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.953632 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06664ca-c52a-4f51-aa10-5ea66c04df7f-serving-certs-ca-bundle\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.954150 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.953901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06664ca-c52a-4f51-aa10-5ea66c04df7f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.955581 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.955554 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.956044 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.956001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-telemeter-client-tls\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.958323 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.958268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-federate-client-tls\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.958323 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.958300 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c06664ca-c52a-4f51-aa10-5ea66c04df7f-secret-telemeter-client\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:18.963858 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:18.963801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76kd\" (UniqueName: \"kubernetes.io/projected/c06664ca-c52a-4f51-aa10-5ea66c04df7f-kube-api-access-x76kd\") pod \"telemeter-client-64fccd8557-gwgwd\" (UID: \"c06664ca-c52a-4f51-aa10-5ea66c04df7f\") " pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:19.052704 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.052668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14753c04-960a-41c7-8111-95101d4dfeba-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-b9n2x\" (UID: \"14753c04-960a-41c7-8111-95101d4dfeba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:19.056291 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.056257 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14753c04-960a-41c7-8111-95101d4dfeba-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-b9n2x\" (UID: \"14753c04-960a-41c7-8111-95101d4dfeba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:19.110198 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.109958 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" Jul 16 17:33:19.293177 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.293111 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:19.346809 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.346722 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64fccd8557-gwgwd"] Jul 16 17:33:19.362490 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:33:19.362459 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06664ca_c52a_4f51_aa10_5ea66c04df7f.slice/crio-d051a5ab349c6d4e971fc23263231fcf7de118e4b6e18dede35a6fbbec2a0d5c WatchSource:0}: Error finding container d051a5ab349c6d4e971fc23263231fcf7de118e4b6e18dede35a6fbbec2a0d5c: Status 404 returned error can't find the container with id d051a5ab349c6d4e971fc23263231fcf7de118e4b6e18dede35a6fbbec2a0d5c Jul 16 17:33:19.522819 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.522789 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x"] Jul 16 17:33:19.526439 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:33:19.526398 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14753c04_960a_41c7_8111_95101d4dfeba.slice/crio-0bc86f7e18e52ff57ad2e9ffaf9e62c068e7e1ae9cd8feb19421889bd7d0179b WatchSource:0}: Error finding container 0bc86f7e18e52ff57ad2e9ffaf9e62c068e7e1ae9cd8feb19421889bd7d0179b: Status 404 returned error can't find the container with id 0bc86f7e18e52ff57ad2e9ffaf9e62c068e7e1ae9cd8feb19421889bd7d0179b Jul 16 17:33:19.572792 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.571796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f79d86fbb-hkrhg" event={"ID":"b1fafa62-69d1-4086-b75d-7b59e92ec22a","Type":"ContainerStarted","Data":"82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f"} Jul 16 17:33:19.572792 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.571887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f79d86fbb-hkrhg" event={"ID":"b1fafa62-69d1-4086-b75d-7b59e92ec22a","Type":"ContainerStarted","Data":"17e5113d8d3e46eb436f3ca344e5458706f6fb115af901b9d902617600e81fee"} Jul 16 17:33:19.575149 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.575121 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" event={"ID":"c06664ca-c52a-4f51-aa10-5ea66c04df7f","Type":"ContainerStarted","Data":"d051a5ab349c6d4e971fc23263231fcf7de118e4b6e18dede35a6fbbec2a0d5c"} Jul 16 17:33:19.577652 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.577611 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" event={"ID":"14753c04-960a-41c7-8111-95101d4dfeba","Type":"ContainerStarted","Data":"0bc86f7e18e52ff57ad2e9ffaf9e62c068e7e1ae9cd8feb19421889bd7d0179b"} Jul 16 17:33:19.582640 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.582610 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwcvg" event={"ID":"68f611a5-8e24-4dbe-a1fd-316365337579","Type":"ContainerStarted","Data":"30b16ac25d613c7e6699070c9aca11da2807b4bb63a705c1951dc10e281dd007"} Jul 16 17:33:19.582758 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.582656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qwcvg" event={"ID":"68f611a5-8e24-4dbe-a1fd-316365337579","Type":"ContainerStarted","Data":"c1080194973c23f4b3f9c7f3d27d735ec35435131e053ab41f0ce08891800da9"} Jul 16 17:33:19.596299 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.596223 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f79d86fbb-hkrhg" podStartSLOduration=1.5961944080000001 podStartE2EDuration="1.596194408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:19.594771832 +0000 UTC m=+180.266242037" watchObservedRunningTime="2026-07-16 17:33:19.596194408 +0000 UTC m=+180.267664619" Jul 16 17:33:19.624888 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:19.623386 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qwcvg" podStartSLOduration=4.464028026 podStartE2EDuration="6.623365438s" totalImagesPullingTime="2.044888504s" totalInitContainerRuntime="114.448908ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:19.621287333 +0000 UTC m=+180.292757538" watchObservedRunningTime="2026-07-16 17:33:19.623365438 +0000 UTC m=+180.294835643" Jul 16 17:33:21.594229 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:21.594198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerStarted","Data":"99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa"} Jul 16 17:33:21.594605 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:21.594263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerStarted","Data":"1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49"} Jul 16 17:33:21.596179 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:21.596136 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" event={"ID":"f3e255e5-d6e8-4fce-a1eb-1db19da83c80","Type":"ContainerStarted","Data":"c0311da14c59c1dbe58f70d9a9aaf9b66dc88d9bd3ffc9bdcc7a8cd9a5b1d03a"} Jul 16 17:33:21.596271 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:21.596210 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" event={"ID":"f3e255e5-d6e8-4fce-a1eb-1db19da83c80","Type":"ContainerStarted","Data":"b3fce0da400a316a120c1cab0e64e7c5957e0c2c2772190e63b9e01eb2ef264c"} Jul 16 17:33:24.611206 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.610924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" event={"ID":"c06664ca-c52a-4f51-aa10-5ea66c04df7f","Type":"ContainerStarted","Data":"a46f5ebae64c12843ac48c70f41e600a982ded643da729b144ab3faf78f07d7c"} Jul 16 17:33:24.611206 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.610996 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" event={"ID":"c06664ca-c52a-4f51-aa10-5ea66c04df7f","Type":"ContainerStarted","Data":"44d963c4278a947dcc52123d53b7661b8a52b8841c5638ad7103c987e8b56f1e"} Jul 16 17:33:24.611206 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.611025 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" event={"ID":"c06664ca-c52a-4f51-aa10-5ea66c04df7f","Type":"ContainerStarted","Data":"7559b8055383950dc544ff3e96315f3afbd43cb8790931c99f54c4cb3c2d84bc"} Jul 16 17:33:24.614597 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.614568 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerStarted","Data":"310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743"} Jul 16 17:33:24.614780 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.614646 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerStarted","Data":"3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343"} Jul 16 17:33:24.614780 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.614691 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerStarted","Data":"0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4"} Jul 16 17:33:24.617191 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.617168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" event={"ID":"f3e255e5-d6e8-4fce-a1eb-1db19da83c80","Type":"ContainerStarted","Data":"37e91c7f7b18fd07ca6744e48cb5d9c2be2b5caa4bec0442ccba90a2297e6ef4"} Jul 16 17:33:24.619313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.618909 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" event={"ID":"14753c04-960a-41c7-8111-95101d4dfeba","Type":"ContainerStarted","Data":"61948883a33c6340fb61f9dc3156e59eab0750d5d27f1ead67c43d81680b2428"} Jul 16 17:33:24.619313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.619022 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:24.620856 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.620806 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" event={"ID":"c3ad62a1-601f-4bb1-acc5-c567cc32f9cb","Type":"ContainerStarted","Data":"65e682c2d0b975afc99df44331d119afd54f76daeb020bc15c777a0b5561d53d"} Jul 16 17:33:24.624715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.624693 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" Jul 16 17:33:24.640941 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.640897 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-64fccd8557-gwgwd" podStartSLOduration=2.430844175 podStartE2EDuration="6.640883553s" totalImagesPullingTime="4.210039378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:24.637812988 +0000 UTC m=+185.309283194" watchObservedRunningTime="2026-07-16 17:33:24.640883553 +0000 UTC m=+185.312353757" Jul 16 17:33:24.662647 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.662598 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-b9n2x" podStartSLOduration=2.620969075 podStartE2EDuration="6.662578736s" totalImagesPullingTime="4.041609661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:24.661734202 +0000 UTC m=+185.333204407" watchObservedRunningTime="2026-07-16 17:33:24.662578736 +0000 UTC m=+185.334048942" Jul 16 17:33:24.687228 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:24.686852 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" podStartSLOduration=2.638643857 podStartE2EDuration="7.686817592s" totalImagesPullingTime="5.048173735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:24.685204366 +0000 UTC m=+185.356674567" watchObservedRunningTime="2026-07-16 17:33:24.686817592 +0000 UTC m=+185.358287795" Jul 16 17:33:25.733435 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.733409 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f79d86fbb-hkrhg"] Jul 16 17:33:25.786663 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.786637 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6647b6996c-nc9zb"] Jul 16 17:33:25.824233 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.824206 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6647b6996c-nc9zb"] Jul 16 17:33:25.824356 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.824338 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.848972 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.848948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-oauth-config\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.849111 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.848976 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-service-ca\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.849111 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.848994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-trusted-ca-bundle\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.849111 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.849025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8k2r\" (UniqueName: \"kubernetes.io/projected/70c43429-2eb8-44f7-8d36-806d762a110c-kube-api-access-v8k2r\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.849111 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.849065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-serving-cert\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.849321 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.849118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-console-config\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.849321 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.849142 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-oauth-serving-cert\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.950355 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.950320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-serving-cert\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.950511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.950388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-console-config\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.950511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.950417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-oauth-serving-cert\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.950511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.950507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-oauth-config\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.950637 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.950529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-service-ca\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.950637 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.950552 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-trusted-ca-bundle\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.950738 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.950717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8k2r\" (UniqueName: \"kubernetes.io/projected/70c43429-2eb8-44f7-8d36-806d762a110c-kube-api-access-v8k2r\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.951383 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.951357 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-console-config\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.951487 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.951377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-service-ca\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.951487 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.951403 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-oauth-serving-cert\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.951668 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.951650 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-trusted-ca-bundle\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.953451 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.953431 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-serving-cert\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.953550 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.953484 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-oauth-config\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:25.962751 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:25.962732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8k2r\" (UniqueName: \"kubernetes.io/projected/70c43429-2eb8-44f7-8d36-806d762a110c-kube-api-access-v8k2r\") pod \"console-6647b6996c-nc9zb\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:26.134330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.134295 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:26.287726 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.287571 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6647b6996c-nc9zb"] Jul 16 17:33:26.290442 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:33:26.290416 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c43429_2eb8_44f7_8d36_806d762a110c.slice/crio-6b1a4ace62fa3ff3ef79bf37e3b4415abbef50405848cb4fc70bd70ffed4d879 WatchSource:0}: Error finding container 6b1a4ace62fa3ff3ef79bf37e3b4415abbef50405848cb4fc70bd70ffed4d879: Status 404 returned error can't find the container with id 6b1a4ace62fa3ff3ef79bf37e3b4415abbef50405848cb4fc70bd70ffed4d879 Jul 16 17:33:26.631310 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.631275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6647b6996c-nc9zb" event={"ID":"70c43429-2eb8-44f7-8d36-806d762a110c","Type":"ContainerStarted","Data":"284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7"} Jul 16 17:33:26.631479 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.631329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6647b6996c-nc9zb" event={"ID":"70c43429-2eb8-44f7-8d36-806d762a110c","Type":"ContainerStarted","Data":"6b1a4ace62fa3ff3ef79bf37e3b4415abbef50405848cb4fc70bd70ffed4d879"} Jul 16 17:33:26.634652 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.634625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerStarted","Data":"478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985"} Jul 16 17:33:26.637554 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.637495 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" event={"ID":"f3e255e5-d6e8-4fce-a1eb-1db19da83c80","Type":"ContainerStarted","Data":"a364029fe522069c184bbc40a2f45875a885428c81321bb2361afe337b0fea2d"} Jul 16 17:33:26.637649 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.637552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" event={"ID":"f3e255e5-d6e8-4fce-a1eb-1db19da83c80","Type":"ContainerStarted","Data":"3fa800c9ef323a139d5e01e0a3a2759cbe2906d4670096b42cf51b06ac1bb9bc"} Jul 16 17:33:26.637649 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.637586 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" event={"ID":"f3e255e5-d6e8-4fce-a1eb-1db19da83c80","Type":"ContainerStarted","Data":"825952aae6bd68f588ddee39dd5bd02c69722da7dd7926c844ada57a9c964caf"} Jul 16 17:33:26.637649 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.637630 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:26.643162 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.643144 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" Jul 16 17:33:26.655257 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.655214 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6647b6996c-nc9zb" podStartSLOduration=1.655201092 podStartE2EDuration="1.655201092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:26.653738191 +0000 UTC m=+187.325208394" watchObservedRunningTime="2026-07-16 17:33:26.655201092 +0000 UTC m=+187.326671297" Jul 16 17:33:26.683682 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.683638 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-f5f78957b-zphtm" podStartSLOduration=5.047292892 podStartE2EDuration="10.683622792s" totalImagesPullingTime="5.6363299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:26.682672123 +0000 UTC m=+187.354142326" watchObservedRunningTime="2026-07-16 17:33:26.683622792 +0000 UTC m=+187.355092997" Jul 16 17:33:26.718910 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:26.718864 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.51214229 podStartE2EDuration="12.718820166s" totalImagesPullingTime="6.176079634s" totalInitContainerRuntime="30.598242ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:26.715641522 +0000 UTC m=+187.387111725" watchObservedRunningTime="2026-07-16 17:33:26.718820166 +0000 UTC m=+187.390290371" Jul 16 17:33:27.565331 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:27.565292 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mspz8" Jul 16 17:33:28.623427 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:28.623394 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:29.652896 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:29.652863 2565 generic.go:356] "Generic (PLEG): container finished" podID="673405af-1260-4611-b0c9-f7ace3907095" containerID="2839c57dcecb6aa6b68e97ee774cae0a562d6285862296adcfc8e52ace6bf851" exitCode=0 Jul 16 17:33:29.653268 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:29.652943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" event={"ID":"673405af-1260-4611-b0c9-f7ace3907095","Type":"ContainerDied","Data":"2839c57dcecb6aa6b68e97ee774cae0a562d6285862296adcfc8e52ace6bf851"} Jul 16 17:33:29.653313 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:29.653279 2565 scope.go:117] "RemoveContainer" containerID="2839c57dcecb6aa6b68e97ee774cae0a562d6285862296adcfc8e52ace6bf851" Jul 16 17:33:29.987452 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:29.987362 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c4cdc6456-wdl4x" podUID="c7bac37f-01ff-4f9d-9157-0eab98e3729f" containerName="console" containerID="cri-o://c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64" gracePeriod=15 Jul 16 17:33:30.281114 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.281088 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:33:30.393637 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.393605 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-config\") pod \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " Jul 16 17:33:30.393829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.393644 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjc7p\" (UniqueName: \"kubernetes.io/projected/c7bac37f-01ff-4f9d-9157-0eab98e3729f-kube-api-access-pjc7p\") pod \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " Jul 16 17:33:30.393829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.393697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-oauth-config\") pod \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " Jul 16 17:33:30.393988 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.393888 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-service-ca\") pod \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " Jul 16 17:33:30.393988 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.393941 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-oauth-serving-cert\") pod \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " Jul 16 17:33:30.394088 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.394007 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-serving-cert\") pod \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\" (UID: \"c7bac37f-01ff-4f9d-9157-0eab98e3729f\") " Jul 16 17:33:30.394140 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.394077 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-config" (OuterVolumeSpecName: "console-config") pod "c7bac37f-01ff-4f9d-9157-0eab98e3729f" (UID: "c7bac37f-01ff-4f9d-9157-0eab98e3729f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:30.394320 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.394294 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-service-ca" (OuterVolumeSpecName: "service-ca") pod "c7bac37f-01ff-4f9d-9157-0eab98e3729f" (UID: "c7bac37f-01ff-4f9d-9157-0eab98e3729f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:30.394390 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.394362 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:30.394518 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.394458 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c7bac37f-01ff-4f9d-9157-0eab98e3729f" (UID: "c7bac37f-01ff-4f9d-9157-0eab98e3729f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:30.396246 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.396219 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c7bac37f-01ff-4f9d-9157-0eab98e3729f" (UID: "c7bac37f-01ff-4f9d-9157-0eab98e3729f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:33:30.396345 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.396249 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bac37f-01ff-4f9d-9157-0eab98e3729f-kube-api-access-pjc7p" (OuterVolumeSpecName: "kube-api-access-pjc7p") pod "c7bac37f-01ff-4f9d-9157-0eab98e3729f" (UID: "c7bac37f-01ff-4f9d-9157-0eab98e3729f"). InnerVolumeSpecName "kube-api-access-pjc7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:30.396385 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.396350 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c7bac37f-01ff-4f9d-9157-0eab98e3729f" (UID: "c7bac37f-01ff-4f9d-9157-0eab98e3729f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:33:30.495552 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.495476 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-service-ca\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:30.495552 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.495503 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7bac37f-01ff-4f9d-9157-0eab98e3729f-oauth-serving-cert\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:30.495552 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.495513 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-serving-cert\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:30.495552 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.495522 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjc7p\" (UniqueName: \"kubernetes.io/projected/c7bac37f-01ff-4f9d-9157-0eab98e3729f-kube-api-access-pjc7p\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:30.495552 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.495530 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7bac37f-01ff-4f9d-9157-0eab98e3729f-console-oauth-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:30.659273 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.659241 2565 generic.go:356] "Generic (PLEG): container finished" podID="c7bac37f-01ff-4f9d-9157-0eab98e3729f" containerID="c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64" exitCode=0 Jul 16 17:33:30.659684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.659287 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4cdc6456-wdl4x" Jul 16 17:33:30.659684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.659326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4cdc6456-wdl4x" event={"ID":"c7bac37f-01ff-4f9d-9157-0eab98e3729f","Type":"ContainerDied","Data":"c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64"} Jul 16 17:33:30.659684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.659402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4cdc6456-wdl4x" event={"ID":"c7bac37f-01ff-4f9d-9157-0eab98e3729f","Type":"ContainerDied","Data":"b555e59a8bc0d9518635410f08d04c896951f5e518b775e2b12b373e5776da17"} Jul 16 17:33:30.659684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.659427 2565 scope.go:117] "RemoveContainer" containerID="c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64" Jul 16 17:33:30.661495 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.661470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mxfhw" event={"ID":"673405af-1260-4611-b0c9-f7ace3907095","Type":"ContainerStarted","Data":"7f5e8ff1c7d47937a97cb52ed1df7979382de818ac753a1960a36dbd876e9c98"} Jul 16 17:33:30.670870 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.670712 2565 scope.go:117] "RemoveContainer" containerID="c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64" Jul 16 17:33:30.671581 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:30.671552 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64\": container with ID starting with c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64 not found: ID does not exist" containerID="c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64" Jul 16 17:33:30.671657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.671595 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64"} err="failed to get container status \"c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64\": rpc error: code = NotFound desc = could not find container \"c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64\": container with ID starting with c838977acada4b6e205c479c83acea8992f866c53b085d00abfa7a42c96e5c64 not found: ID does not exist" Jul 16 17:33:30.704268 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.704244 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4cdc6456-wdl4x"] Jul 16 17:33:30.709798 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:30.709771 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4cdc6456-wdl4x"] Jul 16 17:33:31.877750 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:31.877718 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7bac37f-01ff-4f9d-9157-0eab98e3729f" path="/var/lib/kubelet/pods/c7bac37f-01ff-4f9d-9157-0eab98e3729f/volumes" Jul 16 17:33:36.135354 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:36.135316 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:36.135880 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:36.135383 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:36.140109 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:36.140088 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:36.695000 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:36.694968 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:33:38.239557 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:38.239527 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:38.239962 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:38.239587 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:48.741455 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:48.741373 2565 generic.go:356] "Generic (PLEG): container finished" podID="86d1a4e2-abcd-4c3f-947d-1cd87cca019d" containerID="85d98e2ced0ef3e1d6ab49c30b9ae4dc232346cb1a4cd8032187a17280268c61" exitCode=0 Jul 16 17:33:48.741455 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:48.741431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" event={"ID":"86d1a4e2-abcd-4c3f-947d-1cd87cca019d","Type":"ContainerDied","Data":"85d98e2ced0ef3e1d6ab49c30b9ae4dc232346cb1a4cd8032187a17280268c61"} Jul 16 17:33:48.741929 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:48.741724 2565 scope.go:117] "RemoveContainer" containerID="85d98e2ced0ef3e1d6ab49c30b9ae4dc232346cb1a4cd8032187a17280268c61" Jul 16 17:33:49.750134 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:49.750102 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-mbmr9" event={"ID":"86d1a4e2-abcd-4c3f-947d-1cd87cca019d","Type":"ContainerStarted","Data":"1d6cf33ce0808788d31c64a31c7cb6bf2de7e4aef2a76dd7cb2f209e04825386"} Jul 16 17:33:50.754286 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:50.754244 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f79d86fbb-hkrhg" podUID="b1fafa62-69d1-4086-b75d-7b59e92ec22a" containerName="console" containerID="cri-o://82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f" gracePeriod=15 Jul 16 17:33:51.031194 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.031112 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:51.184322 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.184294 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-oauth-config\") pod \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " Jul 16 17:33:51.184511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.184334 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-config\") pod \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " Jul 16 17:33:51.184511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.184380 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2qtr\" (UniqueName: \"kubernetes.io/projected/b1fafa62-69d1-4086-b75d-7b59e92ec22a-kube-api-access-n2qtr\") pod \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " Jul 16 17:33:51.184511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.184402 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-trusted-ca-bundle\") pod \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " Jul 16 17:33:51.184511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.184423 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-serving-cert\") pod \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " Jul 16 17:33:51.184511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.184438 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-service-ca\") pod \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " Jul 16 17:33:51.184511 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.184456 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-oauth-serving-cert\") pod \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\" (UID: \"b1fafa62-69d1-4086-b75d-7b59e92ec22a\") " Jul 16 17:33:51.184921 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.184890 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-config" (OuterVolumeSpecName: "console-config") pod "b1fafa62-69d1-4086-b75d-7b59e92ec22a" (UID: "b1fafa62-69d1-4086-b75d-7b59e92ec22a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:51.185129 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.185079 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b1fafa62-69d1-4086-b75d-7b59e92ec22a" (UID: "b1fafa62-69d1-4086-b75d-7b59e92ec22a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:51.185129 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.185094 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-service-ca" (OuterVolumeSpecName: "service-ca") pod "b1fafa62-69d1-4086-b75d-7b59e92ec22a" (UID: "b1fafa62-69d1-4086-b75d-7b59e92ec22a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:51.185129 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.185123 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b1fafa62-69d1-4086-b75d-7b59e92ec22a" (UID: "b1fafa62-69d1-4086-b75d-7b59e92ec22a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:51.186786 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.186758 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fafa62-69d1-4086-b75d-7b59e92ec22a-kube-api-access-n2qtr" (OuterVolumeSpecName: "kube-api-access-n2qtr") pod "b1fafa62-69d1-4086-b75d-7b59e92ec22a" (UID: "b1fafa62-69d1-4086-b75d-7b59e92ec22a"). InnerVolumeSpecName "kube-api-access-n2qtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:51.186894 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.186780 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b1fafa62-69d1-4086-b75d-7b59e92ec22a" (UID: "b1fafa62-69d1-4086-b75d-7b59e92ec22a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:33:51.186894 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.186799 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b1fafa62-69d1-4086-b75d-7b59e92ec22a" (UID: "b1fafa62-69d1-4086-b75d-7b59e92ec22a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:33:51.285795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.285708 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qtr\" (UniqueName: \"kubernetes.io/projected/b1fafa62-69d1-4086-b75d-7b59e92ec22a-kube-api-access-n2qtr\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:51.285795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.285741 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-trusted-ca-bundle\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:51.285795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.285750 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-serving-cert\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:51.285795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.285759 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-service-ca\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:51.285795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.285767 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-oauth-serving-cert\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:51.285795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.285776 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-oauth-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:51.285795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.285784 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1fafa62-69d1-4086-b75d-7b59e92ec22a-console-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:33:51.759856 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.759808 2565 generic.go:356] "Generic (PLEG): container finished" podID="b1fafa62-69d1-4086-b75d-7b59e92ec22a" containerID="82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f" exitCode=0 Jul 16 17:33:51.760323 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.759879 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f79d86fbb-hkrhg" Jul 16 17:33:51.760323 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.759878 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f79d86fbb-hkrhg" event={"ID":"b1fafa62-69d1-4086-b75d-7b59e92ec22a","Type":"ContainerDied","Data":"82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f"} Jul 16 17:33:51.760323 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.759994 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f79d86fbb-hkrhg" event={"ID":"b1fafa62-69d1-4086-b75d-7b59e92ec22a","Type":"ContainerDied","Data":"17e5113d8d3e46eb436f3ca344e5458706f6fb115af901b9d902617600e81fee"} Jul 16 17:33:51.760323 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.760015 2565 scope.go:117] "RemoveContainer" containerID="82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f" Jul 16 17:33:51.768726 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.768710 2565 scope.go:117] "RemoveContainer" containerID="82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f" Jul 16 17:33:51.769015 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:33:51.768996 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f\": container with ID starting with 82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f not found: ID does not exist" containerID="82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f" Jul 16 17:33:51.769078 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.769023 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f"} err="failed to get container status \"82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f\": rpc error: code = NotFound desc = could not find container \"82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f\": container with ID starting with 82d6f4b6775a81db147dc3686474e8a2808da498288fcc7dcd01c7a027b7534f not found: ID does not exist" Jul 16 17:33:51.783284 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.783256 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f79d86fbb-hkrhg"] Jul 16 17:33:51.784932 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.784909 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f79d86fbb-hkrhg"] Jul 16 17:33:51.876534 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:51.876504 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1fafa62-69d1-4086-b75d-7b59e92ec22a" path="/var/lib/kubelet/pods/b1fafa62-69d1-4086-b75d-7b59e92ec22a/volumes" Jul 16 17:33:58.244900 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:58.244866 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:33:58.248531 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:33:58.248510 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-597f4687df-q7z9v" Jul 16 17:34:31.741374 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:31.741327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:34:31.743756 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:31.743732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d1fc925-7c4d-4e7a-9bb1-8607d8524be7-metrics-certs\") pod \"network-metrics-daemon-wtj6z\" (UID: \"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7\") " pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:34:31.876673 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:31.876642 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hn8wr\"" Jul 16 17:34:31.884054 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:31.884036 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtj6z" Jul 16 17:34:32.040379 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:32.040355 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtj6z"] Jul 16 17:34:32.043086 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:34:32.043062 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1fc925_7c4d_4e7a_9bb1_8607d8524be7.slice/crio-373c0a4e28545c78dba0b6928e585026f4d1053d08fde1868b4118651daa3a89 WatchSource:0}: Error finding container 373c0a4e28545c78dba0b6928e585026f4d1053d08fde1868b4118651daa3a89: Status 404 returned error can't find the container with id 373c0a4e28545c78dba0b6928e585026f4d1053d08fde1868b4118651daa3a89 Jul 16 17:34:32.929814 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:32.929774 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtj6z" event={"ID":"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7","Type":"ContainerStarted","Data":"373c0a4e28545c78dba0b6928e585026f4d1053d08fde1868b4118651daa3a89"} Jul 16 17:34:33.929978 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.929940 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:34:33.930447 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.930311 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="alertmanager" containerID="cri-o://1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49" gracePeriod=120 Jul 16 17:34:33.930447 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.930367 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy-metric" containerID="cri-o://310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743" gracePeriod=120 Jul 16 17:34:33.930447 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.930390 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="prom-label-proxy" containerID="cri-o://478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985" gracePeriod=120 Jul 16 17:34:33.930447 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.930425 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="config-reloader" containerID="cri-o://99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa" gracePeriod=120 Jul 16 17:34:33.930650 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.930401 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy-web" containerID="cri-o://0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4" gracePeriod=120 Jul 16 17:34:33.930650 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.930472 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy" containerID="cri-o://3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343" gracePeriod=120 Jul 16 17:34:33.946691 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.946650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtj6z" event={"ID":"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7","Type":"ContainerStarted","Data":"2ac284c236b3aa9e85492e47afb4dec9300aefb17dfcea2bc8392448f3bfb463"} Jul 16 17:34:33.946804 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.946786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtj6z" event={"ID":"0d1fc925-7c4d-4e7a-9bb1-8607d8524be7","Type":"ContainerStarted","Data":"8ac2133160b51ffafd2a3c7d8d04043a05c6dad0339ebef112bb9f415bb25dde"} Jul 16 17:34:33.986825 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:33.986758 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wtj6z" podStartSLOduration=252.078527255 podStartE2EDuration="4m12.986739891s" totalImagesPullingTime="908.212636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:33.986098185 +0000 UTC m=+254.657568393" watchObservedRunningTime="2026-07-16 17:34:33.986739891 +0000 UTC m=+254.658210094" Jul 16 17:34:34.953316 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:34.953284 2565 generic.go:356] "Generic (PLEG): container finished" podID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerID="478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985" exitCode=0 Jul 16 17:34:34.953316 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:34.953309 2565 generic.go:356] "Generic (PLEG): container finished" podID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerID="3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343" exitCode=0 Jul 16 17:34:34.953316 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:34.953315 2565 generic.go:356] "Generic (PLEG): container finished" podID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerID="99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa" exitCode=0 Jul 16 17:34:34.953316 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:34.953320 2565 generic.go:356] "Generic (PLEG): container finished" podID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerID="1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49" exitCode=0 Jul 16 17:34:34.953790 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:34.953349 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerDied","Data":"478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985"} Jul 16 17:34:34.953790 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:34.953408 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerDied","Data":"3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343"} Jul 16 17:34:34.953790 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:34.953435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerDied","Data":"99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa"} Jul 16 17:34:34.953790 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:34.953461 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerDied","Data":"1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49"} Jul 16 17:34:35.180811 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.180788 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:35.273806 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.273727 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5xdw\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-kube-api-access-x5xdw\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.273806 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.273774 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-main-tls\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.273806 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.273797 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-main-db\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274102 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.273970 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274102 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274012 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-trusted-ca-bundle\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274102 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274044 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274102 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274074 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-metrics-client-ca\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274299 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274104 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-cluster-tls-config\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274299 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274128 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-config-volume\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274299 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274156 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-web-config\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274299 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274178 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-tls-assets\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274299 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274260 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-web\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.274535 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274302 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-config-out\") pod \"b4383218-5bac-44f6-bc83-387bbe73f29c\" (UID: \"b4383218-5bac-44f6-bc83-387bbe73f29c\") " Jul 16 17:34:35.275728 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274174 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:34:35.275869 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.275827 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:34:35.276483 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.274470 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:34:35.277585 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.277511 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-config-out" (OuterVolumeSpecName: "config-out") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:34:35.277897 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.277832 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:34:35.277990 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.277902 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-config-volume" (OuterVolumeSpecName: "config-volume") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:34:35.278717 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.278678 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-kube-api-access-x5xdw" (OuterVolumeSpecName: "kube-api-access-x5xdw") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "kube-api-access-x5xdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:34:35.279035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.279011 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:34:35.279311 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.279287 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:34:35.279411 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.279225 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:34:35.279674 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.279655 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:34:35.282436 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.282416 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:34:35.289507 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.289487 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-web-config" (OuterVolumeSpecName: "web-config") pod "b4383218-5bac-44f6-bc83-387bbe73f29c" (UID: "b4383218-5bac-44f6-bc83-387bbe73f29c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:34:35.375104 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375074 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-main-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375104 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375101 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-main-db\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375104 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375112 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375122 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375131 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375143 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4383218-5bac-44f6-bc83-387bbe73f29c-metrics-client-ca\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375152 2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-cluster-tls-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375160 2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-config-volume\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375169 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-web-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375176 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-tls-assets\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375183 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4383218-5bac-44f6-bc83-387bbe73f29c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375191 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4383218-5bac-44f6-bc83-387bbe73f29c-config-out\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.375309 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.375198 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5xdw\" (UniqueName: \"kubernetes.io/projected/b4383218-5bac-44f6-bc83-387bbe73f29c-kube-api-access-x5xdw\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:34:35.960672 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.960640 2565 generic.go:356] "Generic (PLEG): container finished" podID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerID="310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743" exitCode=0 Jul 16 17:34:35.960672 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.960664 2565 generic.go:356] "Generic (PLEG): container finished" podID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerID="0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4" exitCode=0 Jul 16 17:34:35.961125 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.960723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerDied","Data":"310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743"} Jul 16 17:34:35.961125 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.960779 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerDied","Data":"0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4"} Jul 16 17:34:35.961125 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.960725 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:35.961125 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.960808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4383218-5bac-44f6-bc83-387bbe73f29c","Type":"ContainerDied","Data":"d73270282a591aa94097a2999f6576d7bc3970fd2f98209000fcd8413e9670c7"} Jul 16 17:34:35.961125 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.960870 2565 scope.go:117] "RemoveContainer" containerID="478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985" Jul 16 17:34:35.969022 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.969003 2565 scope.go:117] "RemoveContainer" containerID="310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743" Jul 16 17:34:35.976212 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.976197 2565 scope.go:117] "RemoveContainer" containerID="3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343" Jul 16 17:34:35.983246 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.983231 2565 scope.go:117] "RemoveContainer" containerID="0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4" Jul 16 17:34:35.986333 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.986302 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:34:35.990697 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.990677 2565 scope.go:117] "RemoveContainer" containerID="99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa" Jul 16 17:34:35.993538 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.993503 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:34:35.999020 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:35.999002 2565 scope.go:117] "RemoveContainer" containerID="1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49" Jul 16 17:34:36.008522 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.008500 2565 scope.go:117] "RemoveContainer" containerID="81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b" Jul 16 17:34:36.015684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.015665 2565 scope.go:117] "RemoveContainer" containerID="478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985" Jul 16 17:34:36.015961 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:34:36.015942 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985\": container with ID starting with 478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985 not found: ID does not exist" containerID="478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985" Jul 16 17:34:36.016007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.015969 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985"} err="failed to get container status \"478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985\": rpc error: code = NotFound desc = could not find container \"478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985\": container with ID starting with 478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985 not found: ID does not exist" Jul 16 17:34:36.016007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.015989 2565 scope.go:117] "RemoveContainer" containerID="310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743" Jul 16 17:34:36.016238 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:34:36.016223 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743\": container with ID starting with 310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743 not found: ID does not exist" containerID="310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743" Jul 16 17:34:36.016289 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.016242 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743"} err="failed to get container status \"310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743\": rpc error: code = NotFound desc = could not find container \"310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743\": container with ID starting with 310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743 not found: ID does not exist" Jul 16 17:34:36.016289 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.016255 2565 scope.go:117] "RemoveContainer" containerID="3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343" Jul 16 17:34:36.016492 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:34:36.016472 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343\": container with ID starting with 3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343 not found: ID does not exist" containerID="3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343" Jul 16 17:34:36.016528 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.016500 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343"} err="failed to get container status \"3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343\": rpc error: code = NotFound desc = could not find container \"3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343\": container with ID starting with 3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343 not found: ID does not exist" Jul 16 17:34:36.016528 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.016522 2565 scope.go:117] "RemoveContainer" containerID="0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4" Jul 16 17:34:36.016733 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:34:36.016718 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4\": container with ID starting with 0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4 not found: ID does not exist" containerID="0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4" Jul 16 17:34:36.016774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.016738 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4"} err="failed to get container status \"0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4\": rpc error: code = NotFound desc = could not find container \"0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4\": container with ID starting with 0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4 not found: ID does not exist" Jul 16 17:34:36.016774 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.016752 2565 scope.go:117] "RemoveContainer" containerID="99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa" Jul 16 17:34:36.017026 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:34:36.017008 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa\": container with ID starting with 99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa not found: ID does not exist" containerID="99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa" Jul 16 17:34:36.017084 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.017031 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa"} err="failed to get container status \"99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa\": rpc error: code = NotFound desc = could not find container \"99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa\": container with ID starting with 99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa not found: ID does not exist" Jul 16 17:34:36.017084 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.017044 2565 scope.go:117] "RemoveContainer" containerID="1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49" Jul 16 17:34:36.017323 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:34:36.017308 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49\": container with ID starting with 1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49 not found: ID does not exist" containerID="1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49" Jul 16 17:34:36.017372 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.017325 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49"} err="failed to get container status \"1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49\": rpc error: code = NotFound desc = could not find container \"1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49\": container with ID starting with 1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49 not found: ID does not exist" Jul 16 17:34:36.017372 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.017349 2565 scope.go:117] "RemoveContainer" containerID="81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b" Jul 16 17:34:36.017636 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:34:36.017619 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b\": container with ID starting with 81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b not found: ID does not exist" containerID="81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b" Jul 16 17:34:36.017688 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.017639 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b"} err="failed to get container status \"81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b\": rpc error: code = NotFound desc = could not find container \"81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b\": container with ID starting with 81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b not found: ID does not exist" Jul 16 17:34:36.017688 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.017653 2565 scope.go:117] "RemoveContainer" containerID="478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985" Jul 16 17:34:36.017962 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.017943 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985"} err="failed to get container status \"478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985\": rpc error: code = NotFound desc = could not find container \"478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985\": container with ID starting with 478499dd8c02f2cde625425ad05bf050fa0f99e006d166ad6244aa6f315fb985 not found: ID does not exist" Jul 16 17:34:36.018018 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.017963 2565 scope.go:117] "RemoveContainer" containerID="310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743" Jul 16 17:34:36.018210 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.018192 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743"} err="failed to get container status \"310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743\": rpc error: code = NotFound desc = could not find container \"310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743\": container with ID starting with 310d1678865db913e002679b622a9b0f6e94ca9368bf4954687d12bee1aee743 not found: ID does not exist" Jul 16 17:34:36.018249 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.018210 2565 scope.go:117] "RemoveContainer" containerID="3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343" Jul 16 17:34:36.018419 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.018399 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343"} err="failed to get container status \"3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343\": rpc error: code = NotFound desc = could not find container \"3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343\": container with ID starting with 3cfb3a930f0c31f39520c8b0f4dcaf45c257d7e412098ff5bbd5ad6160530343 not found: ID does not exist" Jul 16 17:34:36.018470 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.018421 2565 scope.go:117] "RemoveContainer" containerID="0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4" Jul 16 17:34:36.018635 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.018620 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4"} err="failed to get container status \"0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4\": rpc error: code = NotFound desc = could not find container \"0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4\": container with ID starting with 0adf0afa5f7eeba05cd0b3ee100eced9a2da5110c88f28c87e288c0447f11dc4 not found: ID does not exist" Jul 16 17:34:36.018635 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.018634 2565 scope.go:117] "RemoveContainer" containerID="99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa" Jul 16 17:34:36.018825 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.018810 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa"} err="failed to get container status \"99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa\": rpc error: code = NotFound desc = could not find container \"99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa\": container with ID starting with 99a89eeab17edd6362f7242391afb9bbac10ab642cd644fa7e7139a1420bc5fa not found: ID does not exist" Jul 16 17:34:36.018899 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.018825 2565 scope.go:117] "RemoveContainer" containerID="1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49" Jul 16 17:34:36.019028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.019008 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49"} err="failed to get container status \"1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49\": rpc error: code = NotFound desc = could not find container \"1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49\": container with ID starting with 1b06e623717da452d997003efa50337071e522d8b4625be497dd9707873a7b49 not found: ID does not exist" Jul 16 17:34:36.019069 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.019031 2565 scope.go:117] "RemoveContainer" containerID="81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b" Jul 16 17:34:36.019198 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.019181 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b"} err="failed to get container status \"81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b\": rpc error: code = NotFound desc = could not find container \"81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b\": container with ID starting with 81af6728ec7d8748de8579ac348683c46b3440098cbd71831a516977e69f0c3b not found: ID does not exist" Jul 16 17:34:36.029868 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.029743 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:34:36.030211 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030195 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy-web" Jul 16 17:34:36.030293 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030217 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy-web" Jul 16 17:34:36.030293 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030241 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="config-reloader" Jul 16 17:34:36.030293 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030250 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="config-reloader" Jul 16 17:34:36.030293 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030269 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7bac37f-01ff-4f9d-9157-0eab98e3729f" containerName="console" Jul 16 17:34:36.030293 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030277 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7bac37f-01ff-4f9d-9157-0eab98e3729f" containerName="console" Jul 16 17:34:36.030293 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030293 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030302 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030310 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy-metric" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030319 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy-metric" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030329 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="prom-label-proxy" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030337 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="prom-label-proxy" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030347 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="init-config-reloader" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030354 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="init-config-reloader" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030368 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fafa62-69d1-4086-b75d-7b59e92ec22a" containerName="console" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030376 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fafa62-69d1-4086-b75d-7b59e92ec22a" containerName="console" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030390 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="alertmanager" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030398 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="alertmanager" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030475 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="config-reloader" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030489 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="alertmanager" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030497 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy-web" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030506 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="prom-label-proxy" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030515 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7bac37f-01ff-4f9d-9157-0eab98e3729f" containerName="console" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030525 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fafa62-69d1-4086-b75d-7b59e92ec22a" containerName="console" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030534 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy" Jul 16 17:34:36.030573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.030542 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" containerName="kube-rbac-proxy-metric" Jul 16 17:34:36.036132 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.036114 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.040486 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.040459 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 17:34:36.040584 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.040553 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 17:34:36.041032 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.041013 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 17:34:36.041120 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.041041 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 17:34:36.041282 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.041268 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 17:34:36.041407 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.041393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zlcsb\"" Jul 16 17:34:36.043228 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.043212 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 17:34:36.043510 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.043493 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 17:34:36.046137 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.046119 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 17:34:36.048186 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.048166 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 17:34:36.054705 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.054685 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:34:36.181753 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.181636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-web-config\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.181753 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.181741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.181976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.181791 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-config-out\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.181976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.181808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.181976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.181824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.181976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.181882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.181976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.181901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxfxd\" (UniqueName: \"kubernetes.io/projected/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-kube-api-access-qxfxd\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.181976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.181925 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.182155 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.181978 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.182155 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.182011 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.182155 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.182033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.182155 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.182071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.182155 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.182086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283245 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283430 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283249 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283430 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283281 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283430 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283441 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-web-config\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-config-out\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxfxd\" (UniqueName: \"kubernetes.io/projected/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-kube-api-access-qxfxd\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283671 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.283829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.283693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.286540 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.286369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-config-out\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.286680 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.286658 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.286741 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.286680 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.286741 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.286725 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.286871 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.286764 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.287041 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.287017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-web-config\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.287243 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.287204 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.287322 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.287267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.287322 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.287309 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.287420 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.287370 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.289156 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.289140 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.296942 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.296922 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxfxd\" (UniqueName: \"kubernetes.io/projected/bcefc0e9-6263-4dc4-8b32-7596c52ac9ab-kube-api-access-qxfxd\") pod \"alertmanager-main-0\" (UID: \"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.345989 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.345954 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:34:36.492578 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.492549 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:34:36.493990 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:34:36.493958 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcefc0e9_6263_4dc4_8b32_7596c52ac9ab.slice/crio-5b7c2ffd17127babab032d5b401c4b90b4a40c8e526032c365304fd9a3067635 WatchSource:0}: Error finding container 5b7c2ffd17127babab032d5b401c4b90b4a40c8e526032c365304fd9a3067635: Status 404 returned error can't find the container with id 5b7c2ffd17127babab032d5b401c4b90b4a40c8e526032c365304fd9a3067635 Jul 16 17:34:36.971678 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.971643 2565 generic.go:356] "Generic (PLEG): container finished" podID="bcefc0e9-6263-4dc4-8b32-7596c52ac9ab" containerID="1a0262e475d3a956ae77bd4d377974428e03beec1a5535b19c5925f4ab2fb1b2" exitCode=0 Jul 16 17:34:36.972151 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.971690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab","Type":"ContainerDied","Data":"1a0262e475d3a956ae77bd4d377974428e03beec1a5535b19c5925f4ab2fb1b2"} Jul 16 17:34:36.972151 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:36.971915 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab","Type":"ContainerStarted","Data":"5b7c2ffd17127babab032d5b401c4b90b4a40c8e526032c365304fd9a3067635"} Jul 16 17:34:37.877476 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:37.877441 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4383218-5bac-44f6-bc83-387bbe73f29c" path="/var/lib/kubelet/pods/b4383218-5bac-44f6-bc83-387bbe73f29c/volumes" Jul 16 17:34:37.980035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:37.980003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab","Type":"ContainerStarted","Data":"eb89748ae6fac8964d779e8fe2b596236a1f15537e50bd60f8c2114a7de85bc1"} Jul 16 17:34:37.980407 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:37.980067 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab","Type":"ContainerStarted","Data":"3c6e96dd937e14f82d39a99756bbead7c43c6d1fe7385a04fce46fe7fb84f8fd"} Jul 16 17:34:37.980407 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:37.980095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab","Type":"ContainerStarted","Data":"ab51631c14aacfbf93a491fdbd61a5b6ac19fc7fd452be0158aea837d59135ca"} Jul 16 17:34:37.980407 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:37.980121 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab","Type":"ContainerStarted","Data":"2e2a8aa0b3374f186919b4ed8278b5260f616b3d211b3e221ad1c78bab2044ea"} Jul 16 17:34:37.980407 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:37.980144 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab","Type":"ContainerStarted","Data":"2dbe86d5247e38a22f68bfb46e09656f1111463911cb900edfd39d6bf8e023dd"} Jul 16 17:34:37.980407 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:37.980168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcefc0e9-6263-4dc4-8b32-7596c52ac9ab","Type":"ContainerStarted","Data":"a8b08c8d71b3d0ee31b44a496616799acd95287c2eae4ed6e601da0277e3ed96"} Jul 16 17:34:38.011243 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:38.011198 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.9950652290000002 podStartE2EDuration="2.011180891s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.115662ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:38.008441215 +0000 UTC m=+258.679911445" watchObservedRunningTime="2026-07-16 17:34:38.011180891 +0000 UTC m=+258.682651112" Jul 16 17:34:49.419554 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:34:49.419472 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6647b6996c-nc9zb"] Jul 16 17:35:14.440469 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.440399 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6647b6996c-nc9zb" podUID="70c43429-2eb8-44f7-8d36-806d762a110c" containerName="console" containerID="cri-o://284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7" gracePeriod=15 Jul 16 17:35:14.686439 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.686417 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:35:14.810827 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.810801 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-serving-cert\") pod \"70c43429-2eb8-44f7-8d36-806d762a110c\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " Jul 16 17:35:14.811008 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.810860 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-console-config\") pod \"70c43429-2eb8-44f7-8d36-806d762a110c\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " Jul 16 17:35:14.811008 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.810892 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-service-ca\") pod \"70c43429-2eb8-44f7-8d36-806d762a110c\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " Jul 16 17:35:14.811008 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.810940 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8k2r\" (UniqueName: \"kubernetes.io/projected/70c43429-2eb8-44f7-8d36-806d762a110c-kube-api-access-v8k2r\") pod \"70c43429-2eb8-44f7-8d36-806d762a110c\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " Jul 16 17:35:14.811008 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.810972 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-oauth-config\") pod \"70c43429-2eb8-44f7-8d36-806d762a110c\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " Jul 16 17:35:14.811211 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.811095 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-trusted-ca-bundle\") pod \"70c43429-2eb8-44f7-8d36-806d762a110c\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " Jul 16 17:35:14.811211 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.811126 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-oauth-serving-cert\") pod \"70c43429-2eb8-44f7-8d36-806d762a110c\" (UID: \"70c43429-2eb8-44f7-8d36-806d762a110c\") " Jul 16 17:35:14.811317 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.811264 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-console-config" (OuterVolumeSpecName: "console-config") pod "70c43429-2eb8-44f7-8d36-806d762a110c" (UID: "70c43429-2eb8-44f7-8d36-806d762a110c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:35:14.811401 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.811374 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-service-ca" (OuterVolumeSpecName: "service-ca") pod "70c43429-2eb8-44f7-8d36-806d762a110c" (UID: "70c43429-2eb8-44f7-8d36-806d762a110c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:35:14.811537 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.811506 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "70c43429-2eb8-44f7-8d36-806d762a110c" (UID: "70c43429-2eb8-44f7-8d36-806d762a110c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:35:14.811585 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.811570 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-service-ca\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:35:14.811635 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.811593 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-console-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:35:14.811635 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.811600 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "70c43429-2eb8-44f7-8d36-806d762a110c" (UID: "70c43429-2eb8-44f7-8d36-806d762a110c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:35:14.813156 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.813125 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c43429-2eb8-44f7-8d36-806d762a110c-kube-api-access-v8k2r" (OuterVolumeSpecName: "kube-api-access-v8k2r") pod "70c43429-2eb8-44f7-8d36-806d762a110c" (UID: "70c43429-2eb8-44f7-8d36-806d762a110c"). InnerVolumeSpecName "kube-api-access-v8k2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:35:14.813256 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.813188 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "70c43429-2eb8-44f7-8d36-806d762a110c" (UID: "70c43429-2eb8-44f7-8d36-806d762a110c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:35:14.813256 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.813235 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "70c43429-2eb8-44f7-8d36-806d762a110c" (UID: "70c43429-2eb8-44f7-8d36-806d762a110c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:35:14.912087 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.912041 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-oauth-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:35:14.912087 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.912080 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-trusted-ca-bundle\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:35:14.912087 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.912092 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70c43429-2eb8-44f7-8d36-806d762a110c-oauth-serving-cert\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:35:14.912087 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.912100 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70c43429-2eb8-44f7-8d36-806d762a110c-console-serving-cert\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:35:14.912087 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:14.912108 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8k2r\" (UniqueName: \"kubernetes.io/projected/70c43429-2eb8-44f7-8d36-806d762a110c-kube-api-access-v8k2r\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:35:15.125108 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.125016 2565 generic.go:356] "Generic (PLEG): container finished" podID="70c43429-2eb8-44f7-8d36-806d762a110c" containerID="284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7" exitCode=0 Jul 16 17:35:15.125108 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.125064 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6647b6996c-nc9zb" Jul 16 17:35:15.125298 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.125101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6647b6996c-nc9zb" event={"ID":"70c43429-2eb8-44f7-8d36-806d762a110c","Type":"ContainerDied","Data":"284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7"} Jul 16 17:35:15.125298 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.125160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6647b6996c-nc9zb" event={"ID":"70c43429-2eb8-44f7-8d36-806d762a110c","Type":"ContainerDied","Data":"6b1a4ace62fa3ff3ef79bf37e3b4415abbef50405848cb4fc70bd70ffed4d879"} Jul 16 17:35:15.125298 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.125179 2565 scope.go:117] "RemoveContainer" containerID="284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7" Jul 16 17:35:15.134034 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.134018 2565 scope.go:117] "RemoveContainer" containerID="284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7" Jul 16 17:35:15.134273 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:35:15.134258 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7\": container with ID starting with 284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7 not found: ID does not exist" containerID="284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7" Jul 16 17:35:15.134332 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.134285 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7"} err="failed to get container status \"284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7\": rpc error: code = NotFound desc = could not find container \"284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7\": container with ID starting with 284acca60c5903d1f6639c21a5167f11bd99ca787da2269b117efbd2b25d2ab7 not found: ID does not exist" Jul 16 17:35:15.149122 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.149097 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6647b6996c-nc9zb"] Jul 16 17:35:15.151694 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.151665 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6647b6996c-nc9zb"] Jul 16 17:35:15.876961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:15.876933 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c43429-2eb8-44f7-8d36-806d762a110c" path="/var/lib/kubelet/pods/70c43429-2eb8-44f7-8d36-806d762a110c/volumes" Jul 16 17:35:19.773391 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:35:19.773366 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 17:37:13.191665 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.191627 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l"] Jul 16 17:37:13.192098 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.191996 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c43429-2eb8-44f7-8d36-806d762a110c" containerName="console" Jul 16 17:37:13.192098 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.192009 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c43429-2eb8-44f7-8d36-806d762a110c" containerName="console" Jul 16 17:37:13.192098 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.192070 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c43429-2eb8-44f7-8d36-806d762a110c" containerName="console" Jul 16 17:37:13.194947 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.194932 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" Jul 16 17:37:13.197851 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.197816 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 16 17:37:13.197977 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.197888 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 16 17:37:13.198921 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.198906 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-p8gnx\"" Jul 16 17:37:13.218290 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.218266 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l"] Jul 16 17:37:13.293051 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.293007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jzgq\" (UniqueName: \"kubernetes.io/projected/341402da-904c-44a4-b227-dcc85952c9cc-kube-api-access-2jzgq\") pod \"feast-operator-controller-manager-5bbfd9c774-pd76l\" (UID: \"341402da-904c-44a4-b227-dcc85952c9cc\") " pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" Jul 16 17:37:13.393806 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.393767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzgq\" (UniqueName: \"kubernetes.io/projected/341402da-904c-44a4-b227-dcc85952c9cc-kube-api-access-2jzgq\") pod \"feast-operator-controller-manager-5bbfd9c774-pd76l\" (UID: \"341402da-904c-44a4-b227-dcc85952c9cc\") " pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" Jul 16 17:37:13.403942 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.403905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzgq\" (UniqueName: \"kubernetes.io/projected/341402da-904c-44a4-b227-dcc85952c9cc-kube-api-access-2jzgq\") pod \"feast-operator-controller-manager-5bbfd9c774-pd76l\" (UID: \"341402da-904c-44a4-b227-dcc85952c9cc\") " pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" Jul 16 17:37:13.504961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.504786 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" Jul 16 17:37:13.641069 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.641034 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l"] Jul 16 17:37:13.645233 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:37:13.645204 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341402da_904c_44a4_b227_dcc85952c9cc.slice/crio-ce45e513c4ca77aca4a6417b5b9223c734af535a05b2ea888c698d160e0297e9 WatchSource:0}: Error finding container ce45e513c4ca77aca4a6417b5b9223c734af535a05b2ea888c698d160e0297e9: Status 404 returned error can't find the container with id ce45e513c4ca77aca4a6417b5b9223c734af535a05b2ea888c698d160e0297e9 Jul 16 17:37:13.647201 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:13.647186 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:37:14.598131 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:14.598096 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" event={"ID":"341402da-904c-44a4-b227-dcc85952c9cc","Type":"ContainerStarted","Data":"ce45e513c4ca77aca4a6417b5b9223c734af535a05b2ea888c698d160e0297e9"} Jul 16 17:37:16.608776 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:16.608677 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" event={"ID":"341402da-904c-44a4-b227-dcc85952c9cc","Type":"ContainerStarted","Data":"13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c"} Jul 16 17:37:16.609165 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:16.608815 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" Jul 16 17:37:16.628782 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:16.628725 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" podStartSLOduration=0.918934665 podStartE2EDuration="3.628707514s" totalImagesPullingTime="2.709772849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:37:16.627335086 +0000 UTC m=+417.298805293" watchObservedRunningTime="2026-07-16 17:37:16.628707514 +0000 UTC m=+417.300177725" Jul 16 17:37:27.615866 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:27.615822 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" Jul 16 17:37:49.174940 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.174862 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx"] Jul 16 17:37:49.178768 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.178749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.182350 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.182328 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 16 17:37:49.182493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.182358 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 16 17:37:49.182493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.182329 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 16 17:37:49.182493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.182393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 16 17:37:49.182493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.182432 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 16 17:37:49.182493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.182366 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 16 17:37:49.182716 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.182700 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-d4zkg\"" Jul 16 17:37:49.182782 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.182767 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 16 17:37:49.197288 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.197257 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx"] Jul 16 17:37:49.294662 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.294625 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.294830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.294681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-online-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.294830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.294759 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-ui-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.294830 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.294794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9x6d\" (UniqueName: \"kubernetes.io/projected/46ec189c-917c-495c-a490-86b990e0b0c2-kube-api-access-p9x6d\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.294976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.294907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec189c-917c-495c-a490-86b990e0b0c2-feast-data\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.294976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.294934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-registry-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.396082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.396054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-online-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.396249 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.396103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-ui-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.396249 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.396130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9x6d\" (UniqueName: \"kubernetes.io/projected/46ec189c-917c-495c-a490-86b990e0b0c2-kube-api-access-p9x6d\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.396249 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.396172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec189c-917c-495c-a490-86b990e0b0c2-feast-data\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.396249 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.396190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-registry-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.396249 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.396225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.396496 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:37:49.396276 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 16 17:37:49.396496 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:37:49.396295 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 16 17:37:49.396496 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:37:49.396320 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 16 17:37:49.396496 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:37:49.396368 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-ui-tls podName:46ec189c-917c-495c-a490-86b990e0b0c2 nodeName:}" failed. No retries permitted until 2026-07-16 17:37:49.896345283 +0000 UTC m=+450.567815469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-ui-tls") pod "feast-simple-feast-setup-8467cd4dd6-7v7qx" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2") : secret "feast-simple-feast-setup-ui-tls" not found Jul 16 17:37:49.396496 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:37:49.396390 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-registry-tls podName:46ec189c-917c-495c-a490-86b990e0b0c2 nodeName:}" failed. No retries permitted until 2026-07-16 17:37:49.89637958 +0000 UTC m=+450.567849766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-registry-tls") pod "feast-simple-feast-setup-8467cd4dd6-7v7qx" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2") : secret "feast-simple-feast-setup-registry-tls" not found Jul 16 17:37:49.396496 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:37:49.396404 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls podName:46ec189c-917c-495c-a490-86b990e0b0c2 nodeName:}" failed. No retries permitted until 2026-07-16 17:37:49.896396028 +0000 UTC m=+450.567866231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls") pod "feast-simple-feast-setup-8467cd4dd6-7v7qx" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2") : secret "feast-simple-feast-setup-offline-tls" not found Jul 16 17:37:49.396708 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.396581 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec189c-917c-495c-a490-86b990e0b0c2-feast-data\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.398570 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.398552 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-online-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.407619 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.407594 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9x6d\" (UniqueName: \"kubernetes.io/projected/46ec189c-917c-495c-a490-86b990e0b0c2-kube-api-access-p9x6d\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.898944 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.898916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-registry-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.899104 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.898954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.899201 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:37:49.899180 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 16 17:37:49.899240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.899204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-ui-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.899277 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:37:49.899268 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls podName:46ec189c-917c-495c-a490-86b990e0b0c2 nodeName:}" failed. No retries permitted until 2026-07-16 17:37:50.899246434 +0000 UTC m=+451.570716620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls") pod "feast-simple-feast-setup-8467cd4dd6-7v7qx" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2") : secret "feast-simple-feast-setup-offline-tls" not found Jul 16 17:37:49.901417 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.901391 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-registry-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:49.901692 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:49.901676 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-ui-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:50.906937 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:50.906902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:50.909393 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:50.909376 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-7v7qx\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:50.996138 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:50.996107 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:37:51.145087 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:51.145059 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx"] Jul 16 17:37:51.145698 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:37:51.145675 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46ec189c_917c_495c_a490_86b990e0b0c2.slice/crio-215a29c36622731f2d440391dc7f18de25343bc754c8eb84ed22a04f005101f8 WatchSource:0}: Error finding container 215a29c36622731f2d440391dc7f18de25343bc754c8eb84ed22a04f005101f8: Status 404 returned error can't find the container with id 215a29c36622731f2d440391dc7f18de25343bc754c8eb84ed22a04f005101f8 Jul 16 17:37:51.755977 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:37:51.755939 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerStarted","Data":"215a29c36622731f2d440391dc7f18de25343bc754c8eb84ed22a04f005101f8"} Jul 16 17:38:25.909239 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:25.909143 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerStarted","Data":"365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be"} Jul 16 17:38:30.932143 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:30.932064 2565 generic.go:356] "Generic (PLEG): container finished" podID="46ec189c-917c-495c-a490-86b990e0b0c2" containerID="365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be" exitCode=0 Jul 16 17:38:30.932498 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:30.932139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerDied","Data":"365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be"} Jul 16 17:38:31.938292 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:31.938257 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerStarted","Data":"a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070"} Jul 16 17:38:37.968168 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:37.968132 2565 generic.go:356] "Generic (PLEG): container finished" podID="46ec189c-917c-495c-a490-86b990e0b0c2" containerID="a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070" exitCode=0 Jul 16 17:38:37.968576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:37.968204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerDied","Data":"a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070"} Jul 16 17:38:37.982467 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:37.982436 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 16 17:38:38.979912 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.979869 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerStarted","Data":"1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0"} Jul 16 17:38:38.980463 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.979936 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerStarted","Data":"8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3"} Jul 16 17:38:38.980463 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.979957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerStarted","Data":"144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e"} Jul 16 17:38:38.980463 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.979976 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerStarted","Data":"3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278"} Jul 16 17:38:38.996815 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.996783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:38.997107 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.997089 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:38.997275 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.997250 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:38.997399 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.997284 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:38.999118 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.999079 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Jul 16 17:38:38.999232 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.999142 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jul 16 17:38:38.999299 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.999229 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" Jul 16 17:38:38.999353 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:38.999315 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused" Jul 16 17:38:39.013000 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:39.012928 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podStartSLOduration=15.513390818 podStartE2EDuration="50.012908144s" totalImagesPullingTime="34.499517326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:39.010167168 +0000 UTC m=+499.681637373" watchObservedRunningTime="2026-07-16 17:38:39.012908144 +0000 UTC m=+499.684378348" Jul 16 17:38:40.997008 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:40.996972 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:40.997493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:40.997029 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:40.997493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:40.997051 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:40.997493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:40.997068 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:41.996754 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:41.996578 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Jul 16 17:38:41.996754 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:41.996606 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jul 16 17:38:41.996754 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:41.996578 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" Jul 16 17:38:41.996754 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:41.996737 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused" Jul 16 17:38:44.997452 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:44.997420 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:44.997980 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:44.997680 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:44.997980 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:44.997882 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:44.998703 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:44.998680 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:44.998796 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:44.998744 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:44.998988 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:44.998969 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:45.002894 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:45.002873 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:38:45.018784 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:38:45.018759 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:39:46.504482 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.504401 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx"] Jul 16 17:39:46.505055 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.504782 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="registry" containerID="cri-o://3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278" gracePeriod=30 Jul 16 17:39:46.505055 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.504817 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="ui" containerID="cri-o://1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0" gracePeriod=30 Jul 16 17:39:46.505055 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.504817 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="offline" containerID="cri-o://8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3" gracePeriod=30 Jul 16 17:39:46.505055 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.505015 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="online" containerID="cri-o://144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e" gracePeriod=30 Jul 16 17:39:46.806065 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.806033 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx"] Jul 16 17:39:46.810050 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.810032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:46.812911 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.812888 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-7th9l\"" Jul 16 17:39:46.825537 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.825508 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx"] Jul 16 17:39:46.938900 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.938824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-offline-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:46.939082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.938969 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzh7m\" (UniqueName: \"kubernetes.io/projected/791215b5-9831-46b1-a7eb-9d78cfa09b58-kube-api-access-jzh7m\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:46.939082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.939005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-ui-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:46.939082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.939056 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-online-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:46.939244 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.939094 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-registry-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:46.939244 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:46.939167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/791215b5-9831-46b1-a7eb-9d78cfa09b58-feast-data\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.040362 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.040330 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzh7m\" (UniqueName: \"kubernetes.io/projected/791215b5-9831-46b1-a7eb-9d78cfa09b58-kube-api-access-jzh7m\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.040540 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.040378 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-ui-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.040540 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.040443 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-online-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.040540 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.040484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-registry-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.040724 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.040541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/791215b5-9831-46b1-a7eb-9d78cfa09b58-feast-data\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.040724 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.040593 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-offline-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.041158 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.041132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/791215b5-9831-46b1-a7eb-9d78cfa09b58-feast-data\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.043099 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.043076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-online-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.043447 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.043427 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-offline-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.043556 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.043538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-registry-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.043670 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.043650 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-ui-tls\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.058989 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.058909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzh7m\" (UniqueName: \"kubernetes.io/projected/791215b5-9831-46b1-a7eb-9d78cfa09b58-kube-api-access-jzh7m\") pod \"feast-simple-feast-setup-8467cd4dd6-vfctx\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.119894 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.119857 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:39:47.278468 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.278376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx"] Jul 16 17:39:47.280438 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.280409 2565 generic.go:356] "Generic (PLEG): container finished" podID="46ec189c-917c-495c-a490-86b990e0b0c2" containerID="1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0" exitCode=0 Jul 16 17:39:47.280438 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.280437 2565 generic.go:356] "Generic (PLEG): container finished" podID="46ec189c-917c-495c-a490-86b990e0b0c2" containerID="144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e" exitCode=0 Jul 16 17:39:47.280604 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.280459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerDied","Data":"1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0"} Jul 16 17:39:47.280604 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:47.280507 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerDied","Data":"144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e"} Jul 16 17:39:47.280808 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:39:47.280789 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod791215b5_9831_46b1_a7eb_9d78cfa09b58.slice/crio-75a423f42471af19b32b2151fbaea8909c70a13a4db8d0b0f185f1a569b1f319 WatchSource:0}: Error finding container 75a423f42471af19b32b2151fbaea8909c70a13a4db8d0b0f185f1a569b1f319: Status 404 returned error can't find the container with id 75a423f42471af19b32b2151fbaea8909c70a13a4db8d0b0f185f1a569b1f319 Jul 16 17:39:48.286143 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:48.286059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerStarted","Data":"c2623de23d32a8a0c8f0303ca83905e35aea71a50d0e0cbdf503716211a0bc5f"} Jul 16 17:39:48.286143 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:48.286100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerStarted","Data":"75a423f42471af19b32b2151fbaea8909c70a13a4db8d0b0f185f1a569b1f319"} Jul 16 17:39:52.303743 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:52.303713 2565 generic.go:356] "Generic (PLEG): container finished" podID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerID="c2623de23d32a8a0c8f0303ca83905e35aea71a50d0e0cbdf503716211a0bc5f" exitCode=0 Jul 16 17:39:52.304200 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:52.303786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerDied","Data":"c2623de23d32a8a0c8f0303ca83905e35aea71a50d0e0cbdf503716211a0bc5f"} Jul 16 17:39:53.309726 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:53.309690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerStarted","Data":"0fb6906bf50a91f0c3cc21e831db0b405031e9e73ea6581933d81b2e6bea0ea9"} Jul 16 17:39:54.999133 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:54.999086 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Jul 16 17:39:55.016437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:55.016397 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jul 16 17:39:58.332485 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:58.332454 2565 generic.go:356] "Generic (PLEG): container finished" podID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerID="0fb6906bf50a91f0c3cc21e831db0b405031e9e73ea6581933d81b2e6bea0ea9" exitCode=0 Jul 16 17:39:58.332910 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:58.332529 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerDied","Data":"0fb6906bf50a91f0c3cc21e831db0b405031e9e73ea6581933d81b2e6bea0ea9"} Jul 16 17:39:59.343228 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:59.343188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerStarted","Data":"96371b5d3d8ea9cae7c6d043a9b51d113880a6d4980ea366706ec8ee9f540257"} Jul 16 17:39:59.343706 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:59.343248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerStarted","Data":"6bf052f4f06a4430d491ea3bb9fa6340ce41340c2c639d5a5355bc04430d81f6"} Jul 16 17:39:59.343706 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:59.343267 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerStarted","Data":"00ed215e7e134df2182ed1c13bd775a201ede543460e6ae1f29aa31266e2cb6a"} Jul 16 17:39:59.343706 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:59.343283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerStarted","Data":"8b201ecb7689eae5bd1eab25ca73d38082639f227ad5401a7047ee0ec3173947"} Jul 16 17:39:59.376992 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:39:59.376928 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podStartSLOduration=13.376905572 podStartE2EDuration="13.376905572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:39:59.375052261 +0000 UTC m=+580.046522466" watchObservedRunningTime="2026-07-16 17:39:59.376905572 +0000 UTC m=+580.048375777" Jul 16 17:40:02.121027 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:02.120991 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:02.121513 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:02.121058 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:02.121513 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:02.121079 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:02.121513 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:02.121099 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:02.123242 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:02.123204 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.30:8443: connect: connection refused" Jul 16 17:40:02.123242 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:02.123224 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Jul 16 17:40:02.123433 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:02.123295 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.30:8816: connect: connection refused" Jul 16 17:40:02.123568 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:02.123523 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jul 16 17:40:04.998717 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:04.998666 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Jul 16 17:40:05.016694 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.016662 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jul 16 17:40:05.121710 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.121681 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.121915 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.121801 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.122082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.122063 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.122389 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.122371 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.122485 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.122406 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.122485 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.122424 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.122663 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.122637 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.122868 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.122831 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.122960 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.122948 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.126993 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.126974 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.378746 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.378710 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:05.382149 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:05.382124 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:40:14.999314 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:14.999268 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Jul 16 17:40:14.999863 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:14.999431 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:40:15.017043 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:15.017008 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jul 16 17:40:15.017210 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:15.017140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:40:17.161781 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.161758 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:40:17.210646 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.210613 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec189c-917c-495c-a490-86b990e0b0c2-feast-data\") pod \"46ec189c-917c-495c-a490-86b990e0b0c2\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " Jul 16 17:40:17.210798 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.210683 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-registry-tls\") pod \"46ec189c-917c-495c-a490-86b990e0b0c2\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " Jul 16 17:40:17.210798 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.210710 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-online-tls\") pod \"46ec189c-917c-495c-a490-86b990e0b0c2\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " Jul 16 17:40:17.210798 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.210772 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-ui-tls\") pod \"46ec189c-917c-495c-a490-86b990e0b0c2\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " Jul 16 17:40:17.211005 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.210821 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9x6d\" (UniqueName: \"kubernetes.io/projected/46ec189c-917c-495c-a490-86b990e0b0c2-kube-api-access-p9x6d\") pod \"46ec189c-917c-495c-a490-86b990e0b0c2\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " Jul 16 17:40:17.211005 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.210898 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls\") pod \"46ec189c-917c-495c-a490-86b990e0b0c2\" (UID: \"46ec189c-917c-495c-a490-86b990e0b0c2\") " Jul 16 17:40:17.211212 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.211185 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ec189c-917c-495c-a490-86b990e0b0c2-feast-data" (OuterVolumeSpecName: "feast-data") pod "46ec189c-917c-495c-a490-86b990e0b0c2" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:40:17.213454 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.213428 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "46ec189c-917c-495c-a490-86b990e0b0c2" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:40:17.213565 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.213525 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "46ec189c-917c-495c-a490-86b990e0b0c2" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:40:17.213632 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.213590 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ec189c-917c-495c-a490-86b990e0b0c2-kube-api-access-p9x6d" (OuterVolumeSpecName: "kube-api-access-p9x6d") pod "46ec189c-917c-495c-a490-86b990e0b0c2" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2"). InnerVolumeSpecName "kube-api-access-p9x6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:40:17.213632 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.213617 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-online-tls" (OuterVolumeSpecName: "online-tls") pod "46ec189c-917c-495c-a490-86b990e0b0c2" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:40:17.213704 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.213662 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "46ec189c-917c-495c-a490-86b990e0b0c2" (UID: "46ec189c-917c-495c-a490-86b990e0b0c2"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:40:17.312362 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.312334 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9x6d\" (UniqueName: \"kubernetes.io/projected/46ec189c-917c-495c-a490-86b990e0b0c2-kube-api-access-p9x6d\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:40:17.312362 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.312359 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:40:17.312362 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.312368 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ec189c-917c-495c-a490-86b990e0b0c2-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:40:17.312583 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.312376 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:40:17.312583 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.312383 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:40:17.312583 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.312391 2565 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ec189c-917c-495c-a490-86b990e0b0c2-ui-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:40:17.437818 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.437789 2565 generic.go:356] "Generic (PLEG): container finished" podID="46ec189c-917c-495c-a490-86b990e0b0c2" containerID="8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3" exitCode=137 Jul 16 17:40:17.437818 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.437812 2565 generic.go:356] "Generic (PLEG): container finished" podID="46ec189c-917c-495c-a490-86b990e0b0c2" containerID="3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278" exitCode=137 Jul 16 17:40:17.438024 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.437884 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" Jul 16 17:40:17.438024 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.437880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerDied","Data":"8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3"} Jul 16 17:40:17.438024 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.438001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerDied","Data":"3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278"} Jul 16 17:40:17.438024 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.438017 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx" event={"ID":"46ec189c-917c-495c-a490-86b990e0b0c2","Type":"ContainerDied","Data":"215a29c36622731f2d440391dc7f18de25343bc754c8eb84ed22a04f005101f8"} Jul 16 17:40:17.438154 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.438036 2565 scope.go:117] "RemoveContainer" containerID="1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0" Jul 16 17:40:17.447647 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.447628 2565 scope.go:117] "RemoveContainer" containerID="8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3" Jul 16 17:40:17.455606 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.455583 2565 scope.go:117] "RemoveContainer" containerID="144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e" Jul 16 17:40:17.462712 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.462690 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx"] Jul 16 17:40:17.464253 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.464237 2565 scope.go:117] "RemoveContainer" containerID="3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278" Jul 16 17:40:17.469484 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.469463 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-7v7qx"] Jul 16 17:40:17.472537 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.472524 2565 scope.go:117] "RemoveContainer" containerID="a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070" Jul 16 17:40:17.479993 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.479971 2565 scope.go:117] "RemoveContainer" containerID="365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be" Jul 16 17:40:17.487375 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.487349 2565 scope.go:117] "RemoveContainer" containerID="1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0" Jul 16 17:40:17.487622 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:40:17.487602 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0\": container with ID starting with 1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0 not found: ID does not exist" containerID="1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0" Jul 16 17:40:17.487689 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.487636 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0"} err="failed to get container status \"1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0\": rpc error: code = NotFound desc = could not find container \"1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0\": container with ID starting with 1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0 not found: ID does not exist" Jul 16 17:40:17.487689 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.487664 2565 scope.go:117] "RemoveContainer" containerID="8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3" Jul 16 17:40:17.487927 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:40:17.487912 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3\": container with ID starting with 8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3 not found: ID does not exist" containerID="8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3" Jul 16 17:40:17.487974 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.487942 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3"} err="failed to get container status \"8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3\": rpc error: code = NotFound desc = could not find container \"8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3\": container with ID starting with 8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3 not found: ID does not exist" Jul 16 17:40:17.487974 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.487958 2565 scope.go:117] "RemoveContainer" containerID="144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e" Jul 16 17:40:17.488188 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:40:17.488170 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e\": container with ID starting with 144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e not found: ID does not exist" containerID="144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e" Jul 16 17:40:17.488231 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.488193 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e"} err="failed to get container status \"144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e\": rpc error: code = NotFound desc = could not find container \"144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e\": container with ID starting with 144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e not found: ID does not exist" Jul 16 17:40:17.488231 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.488208 2565 scope.go:117] "RemoveContainer" containerID="3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278" Jul 16 17:40:17.488427 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:40:17.488412 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278\": container with ID starting with 3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278 not found: ID does not exist" containerID="3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278" Jul 16 17:40:17.488488 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.488434 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278"} err="failed to get container status \"3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278\": rpc error: code = NotFound desc = could not find container \"3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278\": container with ID starting with 3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278 not found: ID does not exist" Jul 16 17:40:17.488488 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.488453 2565 scope.go:117] "RemoveContainer" containerID="a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070" Jul 16 17:40:17.488682 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:40:17.488668 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070\": container with ID starting with a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070 not found: ID does not exist" containerID="a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070" Jul 16 17:40:17.488724 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.488685 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070"} err="failed to get container status \"a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070\": rpc error: code = NotFound desc = could not find container \"a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070\": container with ID starting with a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070 not found: ID does not exist" Jul 16 17:40:17.488724 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.488707 2565 scope.go:117] "RemoveContainer" containerID="365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be" Jul 16 17:40:17.488956 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:40:17.488939 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be\": container with ID starting with 365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be not found: ID does not exist" containerID="365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be" Jul 16 17:40:17.489006 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.488968 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be"} err="failed to get container status \"365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be\": rpc error: code = NotFound desc = could not find container \"365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be\": container with ID starting with 365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be not found: ID does not exist" Jul 16 17:40:17.489006 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.488982 2565 scope.go:117] "RemoveContainer" containerID="1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0" Jul 16 17:40:17.489216 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.489196 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0"} err="failed to get container status \"1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0\": rpc error: code = NotFound desc = could not find container \"1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0\": container with ID starting with 1854b97e9fe860ab94bfda779e63b45f286d12816a6ca688cac49d15936fb6e0 not found: ID does not exist" Jul 16 17:40:17.489288 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.489218 2565 scope.go:117] "RemoveContainer" containerID="8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3" Jul 16 17:40:17.489459 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.489441 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3"} err="failed to get container status \"8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3\": rpc error: code = NotFound desc = could not find container \"8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3\": container with ID starting with 8092b191399db6a7bdcebcfecf50bd8c44426cdd2dcd54d3065787f1033f6cc3 not found: ID does not exist" Jul 16 17:40:17.489504 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.489461 2565 scope.go:117] "RemoveContainer" containerID="144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e" Jul 16 17:40:17.489653 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.489637 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e"} err="failed to get container status \"144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e\": rpc error: code = NotFound desc = could not find container \"144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e\": container with ID starting with 144eb7ec1a5240dc484d325aa302863cd016694aa43a7346aba3b5d2fbc6f87e not found: ID does not exist" Jul 16 17:40:17.489690 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.489654 2565 scope.go:117] "RemoveContainer" containerID="3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278" Jul 16 17:40:17.489908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.489882 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278"} err="failed to get container status \"3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278\": rpc error: code = NotFound desc = could not find container \"3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278\": container with ID starting with 3b6ecd373f0cf8796979f5ed4638b012ad27251b9cfbd3ba77ef308ab0596278 not found: ID does not exist" Jul 16 17:40:17.489971 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.489908 2565 scope.go:117] "RemoveContainer" containerID="a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070" Jul 16 17:40:17.490148 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.490130 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070"} err="failed to get container status \"a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070\": rpc error: code = NotFound desc = could not find container \"a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070\": container with ID starting with a744cee4b6a05b61c1894efdba43af64e99e2fd526854955b704aebefd7bd070 not found: ID does not exist" Jul 16 17:40:17.490194 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.490149 2565 scope.go:117] "RemoveContainer" containerID="365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be" Jul 16 17:40:17.490341 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.490325 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be"} err="failed to get container status \"365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be\": rpc error: code = NotFound desc = could not find container \"365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be\": container with ID starting with 365e59cd4b367b4edde45190c15b0267f2b4706c1b274109028228e17fdcd3be not found: ID does not exist" Jul 16 17:40:17.876969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:40:17.876938 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" path="/var/lib/kubelet/pods/46ec189c-917c-495c-a490-86b990e0b0c2/volumes" Jul 16 17:41:08.589684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.589647 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd"] Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590013 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="ui" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590025 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="ui" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590036 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="feast-init" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590041 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="feast-init" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590048 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="online" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590054 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="online" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590061 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="offline" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590065 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="offline" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590074 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="registry" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590079 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="registry" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590086 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="feast-apply" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590091 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="feast-apply" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590144 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="ui" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590152 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="registry" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590159 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="online" Jul 16 17:41:08.590167 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.590167 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ec189c-917c-495c-a490-86b990e0b0c2" containerName="offline" Jul 16 17:41:08.593700 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.593682 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.599223 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.599201 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 16 17:41:08.599993 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.599968 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 16 17:41:08.600107 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.599989 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 16 17:41:08.600605 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.600577 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 16 17:41:08.600833 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.600804 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 16 17:41:08.600833 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.600812 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 16 17:41:08.601019 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.600826 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-gw6bl\"" Jul 16 17:41:08.607427 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.607403 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 16 17:41:08.619157 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.619134 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd"] Jul 16 17:41:08.650517 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.650488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d0d485d8-bc58-4c29-b824-c263f4298c31-registry-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.650682 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.650612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-online-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.650743 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.650690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-ui-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.650743 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.650720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-offline-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.650864 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.650744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d485d8-bc58-4c29-b824-c263f4298c31-feast-data\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.650864 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.650807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ps6v\" (UniqueName: \"kubernetes.io/projected/d0d485d8-bc58-4c29-b824-c263f4298c31-kube-api-access-9ps6v\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.752061 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.752028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d0d485d8-bc58-4c29-b824-c263f4298c31-registry-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.752243 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.752083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-online-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.752243 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.752124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-ui-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.752243 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.752144 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-offline-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.752243 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.752170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d485d8-bc58-4c29-b824-c263f4298c31-feast-data\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.752243 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.752202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ps6v\" (UniqueName: \"kubernetes.io/projected/d0d485d8-bc58-4c29-b824-c263f4298c31-kube-api-access-9ps6v\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.752599 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.752578 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d485d8-bc58-4c29-b824-c263f4298c31-feast-data\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.752775 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.752751 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d0d485d8-bc58-4c29-b824-c263f4298c31-registry-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.754735 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.754707 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-online-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.754818 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.754753 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-ui-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.754818 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.754765 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-offline-tls\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.761608 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.761584 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ps6v\" (UniqueName: \"kubernetes.io/projected/d0d485d8-bc58-4c29-b824-c263f4298c31-kube-api-access-9ps6v\") pod \"feast-simple-feast-remote-setup-db887794b-fgzwd\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:08.903729 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:08.903638 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:09.052417 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:09.052388 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd"] Jul 16 17:41:09.055073 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:41:09.055029 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0d485d8_bc58_4c29_b824_c263f4298c31.slice/crio-fd7609e6e8d5d3c97570dc80429526e210154224b9efe98d6e834a5a202a65a6 WatchSource:0}: Error finding container fd7609e6e8d5d3c97570dc80429526e210154224b9efe98d6e834a5a202a65a6: Status 404 returned error can't find the container with id fd7609e6e8d5d3c97570dc80429526e210154224b9efe98d6e834a5a202a65a6 Jul 16 17:41:09.663219 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:09.663169 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerStarted","Data":"cb2d347573748ae235399890007ff685bc4784701491428e4fe0d6a49ae06b7d"} Jul 16 17:41:09.663219 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:09.663225 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerStarted","Data":"fd7609e6e8d5d3c97570dc80429526e210154224b9efe98d6e834a5a202a65a6"} Jul 16 17:41:14.687069 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:14.687040 2565 generic.go:356] "Generic (PLEG): container finished" podID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerID="cb2d347573748ae235399890007ff685bc4784701491428e4fe0d6a49ae06b7d" exitCode=0 Jul 16 17:41:14.687447 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:14.687122 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerDied","Data":"cb2d347573748ae235399890007ff685bc4784701491428e4fe0d6a49ae06b7d"} Jul 16 17:41:15.693445 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:15.693409 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerStarted","Data":"99234e7028de33f11eec9c580b573a75fdbd99aa2534235155b2f7726a4aec90"} Jul 16 17:41:20.716241 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:20.716210 2565 generic.go:356] "Generic (PLEG): container finished" podID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerID="99234e7028de33f11eec9c580b573a75fdbd99aa2534235155b2f7726a4aec90" exitCode=0 Jul 16 17:41:20.716591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:20.716281 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerDied","Data":"99234e7028de33f11eec9c580b573a75fdbd99aa2534235155b2f7726a4aec90"} Jul 16 17:41:20.720007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:20.719988 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 16 17:41:21.724721 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:21.724682 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerStarted","Data":"bb1694a69b576c966676b9fef12f4964783fa4886a387536b320562fbd1dd057"} Jul 16 17:41:21.725226 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:21.724749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerStarted","Data":"43c4f5ff4300bb8d875a0e7e2808706e93da421175482ac6c481c2f569aa4831"} Jul 16 17:41:21.725226 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:21.724764 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerStarted","Data":"9bb8def803901e2405b0aae824448fa8e21253ae44953107e1b32616ef0184d5"} Jul 16 17:41:21.757619 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:21.757568 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podStartSLOduration=13.757552944 podStartE2EDuration="13.757552944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:21.756234093 +0000 UTC m=+662.427704301" watchObservedRunningTime="2026-07-16 17:41:21.757552944 +0000 UTC m=+662.429023147" Jul 16 17:41:23.904831 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:23.904795 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:23.905414 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:23.905387 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:23.905414 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:23.905419 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:23.906915 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:23.906883 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.31:8816: connect: connection refused" Jul 16 17:41:23.907031 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:23.906915 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused" Jul 16 17:41:23.907031 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:23.906933 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jul 16 17:41:26.905116 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:26.905085 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:26.905586 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:26.905252 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:26.905762 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:26.905736 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:26.905829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:26.905782 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:26.905995 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:26.905975 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:26.906205 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:26.906188 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:26.911746 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:26.911725 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:27.766042 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:27.766006 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:41:27.770587 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:41:27.770564 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:42:29.902345 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:29.902260 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd"] Jul 16 17:42:29.902905 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:29.902570 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="online" containerID="cri-o://9bb8def803901e2405b0aae824448fa8e21253ae44953107e1b32616ef0184d5" gracePeriod=30 Jul 16 17:42:29.902993 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:29.902966 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="offline" containerID="cri-o://43c4f5ff4300bb8d875a0e7e2808706e93da421175482ac6c481c2f569aa4831" gracePeriod=30 Jul 16 17:42:29.903051 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:29.902968 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="ui" containerID="cri-o://bb1694a69b576c966676b9fef12f4964783fa4886a387536b320562fbd1dd057" gracePeriod=30 Jul 16 17:42:30.053828 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:30.053795 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx"] Jul 16 17:42:30.054126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:30.054098 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="registry" containerID="cri-o://8b201ecb7689eae5bd1eab25ca73d38082639f227ad5401a7047ee0ec3173947" gracePeriod=30 Jul 16 17:42:30.054360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:30.054181 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="online" containerID="cri-o://00ed215e7e134df2182ed1c13bd775a201ede543460e6ae1f29aa31266e2cb6a" gracePeriod=30 Jul 16 17:42:30.054454 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:30.054177 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="offline" containerID="cri-o://6bf052f4f06a4430d491ea3bb9fa6340ce41340c2c639d5a5355bc04430d81f6" gracePeriod=30 Jul 16 17:42:30.054454 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:30.054187 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="ui" containerID="cri-o://96371b5d3d8ea9cae7c6d043a9b51d113880a6d4980ea366706ec8ee9f540257" gracePeriod=30 Jul 16 17:42:31.037741 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:31.037665 2565 generic.go:356] "Generic (PLEG): container finished" podID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerID="96371b5d3d8ea9cae7c6d043a9b51d113880a6d4980ea366706ec8ee9f540257" exitCode=0 Jul 16 17:42:31.037741 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:31.037691 2565 generic.go:356] "Generic (PLEG): container finished" podID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerID="00ed215e7e134df2182ed1c13bd775a201ede543460e6ae1f29aa31266e2cb6a" exitCode=0 Jul 16 17:42:31.038194 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:31.037733 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerDied","Data":"96371b5d3d8ea9cae7c6d043a9b51d113880a6d4980ea366706ec8ee9f540257"} Jul 16 17:42:31.038194 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:31.037774 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerDied","Data":"00ed215e7e134df2182ed1c13bd775a201ede543460e6ae1f29aa31266e2cb6a"} Jul 16 17:42:31.040231 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:31.040210 2565 generic.go:356] "Generic (PLEG): container finished" podID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerID="bb1694a69b576c966676b9fef12f4964783fa4886a387536b320562fbd1dd057" exitCode=0 Jul 16 17:42:31.040231 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:31.040228 2565 generic.go:356] "Generic (PLEG): container finished" podID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerID="9bb8def803901e2405b0aae824448fa8e21253ae44953107e1b32616ef0184d5" exitCode=0 Jul 16 17:42:31.040354 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:31.040275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerDied","Data":"bb1694a69b576c966676b9fef12f4964783fa4886a387536b320562fbd1dd057"} Jul 16 17:42:31.040354 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:31.040306 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerDied","Data":"9bb8def803901e2405b0aae824448fa8e21253ae44953107e1b32616ef0184d5"} Jul 16 17:42:35.122631 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:35.122583 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.30:8443: connect: connection refused" Jul 16 17:42:35.379690 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:35.379600 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jul 16 17:42:36.906196 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:36.906150 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused" Jul 16 17:42:37.766270 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:37.766231 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jul 16 17:42:45.123287 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:45.123230 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.30:8443: connect: connection refused" Jul 16 17:42:45.379825 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:45.379733 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jul 16 17:42:46.905995 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:46.905952 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused" Jul 16 17:42:47.767137 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:47.767089 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jul 16 17:42:55.123242 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:55.123193 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.30:8443: connect: connection refused" Jul 16 17:42:55.123664 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:55.123362 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:42:55.379206 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:55.379102 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jul 16 17:42:55.379350 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:55.379222 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:42:56.906124 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:56.906078 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused" Jul 16 17:42:56.906574 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:56.906200 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:42:57.766732 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:57.766682 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jul 16 17:42:57.766957 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:42:57.766881 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:43:00.171482 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.171401 2565 generic.go:356] "Generic (PLEG): container finished" podID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerID="43c4f5ff4300bb8d875a0e7e2808706e93da421175482ac6c481c2f569aa4831" exitCode=137 Jul 16 17:43:00.171482 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.171469 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerDied","Data":"43c4f5ff4300bb8d875a0e7e2808706e93da421175482ac6c481c2f569aa4831"} Jul 16 17:43:00.171924 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.171508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" event={"ID":"d0d485d8-bc58-4c29-b824-c263f4298c31","Type":"ContainerDied","Data":"fd7609e6e8d5d3c97570dc80429526e210154224b9efe98d6e834a5a202a65a6"} Jul 16 17:43:00.171924 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.171520 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd7609e6e8d5d3c97570dc80429526e210154224b9efe98d6e834a5a202a65a6" Jul 16 17:43:00.174247 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.174229 2565 generic.go:356] "Generic (PLEG): container finished" podID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerID="6bf052f4f06a4430d491ea3bb9fa6340ce41340c2c639d5a5355bc04430d81f6" exitCode=137 Jul 16 17:43:00.174247 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.174245 2565 generic.go:356] "Generic (PLEG): container finished" podID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerID="8b201ecb7689eae5bd1eab25ca73d38082639f227ad5401a7047ee0ec3173947" exitCode=137 Jul 16 17:43:00.174401 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.174283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerDied","Data":"6bf052f4f06a4430d491ea3bb9fa6340ce41340c2c639d5a5355bc04430d81f6"} Jul 16 17:43:00.174401 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.174306 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerDied","Data":"8b201ecb7689eae5bd1eab25ca73d38082639f227ad5401a7047ee0ec3173947"} Jul 16 17:43:00.194949 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.194931 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:43:00.235240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.235213 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d0d485d8-bc58-4c29-b824-c263f4298c31-registry-tls\") pod \"d0d485d8-bc58-4c29-b824-c263f4298c31\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " Jul 16 17:43:00.235394 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.235250 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-offline-tls\") pod \"d0d485d8-bc58-4c29-b824-c263f4298c31\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " Jul 16 17:43:00.235394 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.235302 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-ui-tls\") pod \"d0d485d8-bc58-4c29-b824-c263f4298c31\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " Jul 16 17:43:00.235394 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.235326 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-online-tls\") pod \"d0d485d8-bc58-4c29-b824-c263f4298c31\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " Jul 16 17:43:00.235394 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.235343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ps6v\" (UniqueName: \"kubernetes.io/projected/d0d485d8-bc58-4c29-b824-c263f4298c31-kube-api-access-9ps6v\") pod \"d0d485d8-bc58-4c29-b824-c263f4298c31\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " Jul 16 17:43:00.235653 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.235624 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d485d8-bc58-4c29-b824-c263f4298c31-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d0d485d8-bc58-4c29-b824-c263f4298c31" (UID: "d0d485d8-bc58-4c29-b824-c263f4298c31"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:43:00.237716 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.237692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d0d485d8-bc58-4c29-b824-c263f4298c31" (UID: "d0d485d8-bc58-4c29-b824-c263f4298c31"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:43:00.237816 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.237714 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d485d8-bc58-4c29-b824-c263f4298c31-kube-api-access-9ps6v" (OuterVolumeSpecName: "kube-api-access-9ps6v") pod "d0d485d8-bc58-4c29-b824-c263f4298c31" (UID: "d0d485d8-bc58-4c29-b824-c263f4298c31"). InnerVolumeSpecName "kube-api-access-9ps6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:43:00.237816 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.237735 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-online-tls" (OuterVolumeSpecName: "online-tls") pod "d0d485d8-bc58-4c29-b824-c263f4298c31" (UID: "d0d485d8-bc58-4c29-b824-c263f4298c31"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:43:00.237972 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.237956 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d0d485d8-bc58-4c29-b824-c263f4298c31" (UID: "d0d485d8-bc58-4c29-b824-c263f4298c31"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:43:00.336637 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.336602 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d485d8-bc58-4c29-b824-c263f4298c31-feast-data\") pod \"d0d485d8-bc58-4c29-b824-c263f4298c31\" (UID: \"d0d485d8-bc58-4c29-b824-c263f4298c31\") " Jul 16 17:43:00.336867 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.336824 2565 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-ui-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.336917 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.336871 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.336917 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.336887 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ps6v\" (UniqueName: \"kubernetes.io/projected/d0d485d8-bc58-4c29-b824-c263f4298c31-kube-api-access-9ps6v\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.336917 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.336900 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d0d485d8-bc58-4c29-b824-c263f4298c31-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.337008 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.336915 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d0d485d8-bc58-4c29-b824-c263f4298c31-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.337121 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.337104 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d485d8-bc58-4c29-b824-c263f4298c31-feast-data" (OuterVolumeSpecName: "feast-data") pod "d0d485d8-bc58-4c29-b824-c263f4298c31" (UID: "d0d485d8-bc58-4c29-b824-c263f4298c31"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:43:00.437899 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.437811 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d485d8-bc58-4c29-b824-c263f4298c31-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.700336 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.700314 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:43:00.740232 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.740202 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/791215b5-9831-46b1-a7eb-9d78cfa09b58-feast-data\") pod \"791215b5-9831-46b1-a7eb-9d78cfa09b58\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " Jul 16 17:43:00.740371 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.740283 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-offline-tls\") pod \"791215b5-9831-46b1-a7eb-9d78cfa09b58\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " Jul 16 17:43:00.740371 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.740327 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-online-tls\") pod \"791215b5-9831-46b1-a7eb-9d78cfa09b58\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " Jul 16 17:43:00.740371 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.740354 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzh7m\" (UniqueName: \"kubernetes.io/projected/791215b5-9831-46b1-a7eb-9d78cfa09b58-kube-api-access-jzh7m\") pod \"791215b5-9831-46b1-a7eb-9d78cfa09b58\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " Jul 16 17:43:00.740527 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.740405 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-ui-tls\") pod \"791215b5-9831-46b1-a7eb-9d78cfa09b58\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " Jul 16 17:43:00.740527 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.740443 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-registry-tls\") pod \"791215b5-9831-46b1-a7eb-9d78cfa09b58\" (UID: \"791215b5-9831-46b1-a7eb-9d78cfa09b58\") " Jul 16 17:43:00.740745 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.740712 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791215b5-9831-46b1-a7eb-9d78cfa09b58-feast-data" (OuterVolumeSpecName: "feast-data") pod "791215b5-9831-46b1-a7eb-9d78cfa09b58" (UID: "791215b5-9831-46b1-a7eb-9d78cfa09b58"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:43:00.742576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.742549 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791215b5-9831-46b1-a7eb-9d78cfa09b58-kube-api-access-jzh7m" (OuterVolumeSpecName: "kube-api-access-jzh7m") pod "791215b5-9831-46b1-a7eb-9d78cfa09b58" (UID: "791215b5-9831-46b1-a7eb-9d78cfa09b58"). InnerVolumeSpecName "kube-api-access-jzh7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:43:00.742920 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.742903 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "791215b5-9831-46b1-a7eb-9d78cfa09b58" (UID: "791215b5-9831-46b1-a7eb-9d78cfa09b58"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:43:00.743012 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.742953 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "791215b5-9831-46b1-a7eb-9d78cfa09b58" (UID: "791215b5-9831-46b1-a7eb-9d78cfa09b58"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:43:00.743172 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.743147 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-online-tls" (OuterVolumeSpecName: "online-tls") pod "791215b5-9831-46b1-a7eb-9d78cfa09b58" (UID: "791215b5-9831-46b1-a7eb-9d78cfa09b58"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:43:00.743250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.743177 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "791215b5-9831-46b1-a7eb-9d78cfa09b58" (UID: "791215b5-9831-46b1-a7eb-9d78cfa09b58"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:43:00.841785 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.841742 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.841785 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.841779 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzh7m\" (UniqueName: \"kubernetes.io/projected/791215b5-9831-46b1-a7eb-9d78cfa09b58-kube-api-access-jzh7m\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.841785 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.841790 2565 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-ui-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.841785 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.841799 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.842049 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.841807 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/791215b5-9831-46b1-a7eb-9d78cfa09b58-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:00.842049 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:00.841815 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/791215b5-9831-46b1-a7eb-9d78cfa09b58-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:43:01.181978 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.181939 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" event={"ID":"791215b5-9831-46b1-a7eb-9d78cfa09b58","Type":"ContainerDied","Data":"75a423f42471af19b32b2151fbaea8909c70a13a4db8d0b0f185f1a569b1f319"} Jul 16 17:43:01.181978 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.181980 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx" Jul 16 17:43:01.182455 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.182001 2565 scope.go:117] "RemoveContainer" containerID="96371b5d3d8ea9cae7c6d043a9b51d113880a6d4980ea366706ec8ee9f540257" Jul 16 17:43:01.182455 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.181981 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd" Jul 16 17:43:01.191819 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.191803 2565 scope.go:117] "RemoveContainer" containerID="6bf052f4f06a4430d491ea3bb9fa6340ce41340c2c639d5a5355bc04430d81f6" Jul 16 17:43:01.199918 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.199888 2565 scope.go:117] "RemoveContainer" containerID="00ed215e7e134df2182ed1c13bd775a201ede543460e6ae1f29aa31266e2cb6a" Jul 16 17:43:01.207545 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.207528 2565 scope.go:117] "RemoveContainer" containerID="8b201ecb7689eae5bd1eab25ca73d38082639f227ad5401a7047ee0ec3173947" Jul 16 17:43:01.212605 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.212587 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx"] Jul 16 17:43:01.215565 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.215547 2565 scope.go:117] "RemoveContainer" containerID="0fb6906bf50a91f0c3cc21e831db0b405031e9e73ea6581933d81b2e6bea0ea9" Jul 16 17:43:01.220486 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.220467 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8467cd4dd6-vfctx"] Jul 16 17:43:01.222975 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.222955 2565 scope.go:117] "RemoveContainer" containerID="c2623de23d32a8a0c8f0303ca83905e35aea71a50d0e0cbdf503716211a0bc5f" Jul 16 17:43:01.240784 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.240764 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd"] Jul 16 17:43:01.244038 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.244018 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db887794b-fgzwd"] Jul 16 17:43:01.877232 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.877201 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" path="/var/lib/kubelet/pods/791215b5-9831-46b1-a7eb-9d78cfa09b58/volumes" Jul 16 17:43:01.877789 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:01.877775 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" path="/var/lib/kubelet/pods/d0d485d8-bc58-4c29-b824-c263f4298c31/volumes" Jul 16 17:43:12.245798 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.245770 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-m8tc5"] Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246107 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="feast-apply" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246122 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="feast-apply" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246145 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="offline" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246152 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="offline" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246163 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="feast-apply" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246168 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="feast-apply" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246175 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="online" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246183 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="online" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246194 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="ui" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246201 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="ui" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246208 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="ui" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246216 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="ui" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246229 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="registry" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246237 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="registry" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246244 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="online" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246251 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="online" Jul 16 17:43:12.246258 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246266 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="feast-init" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246273 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="feast-init" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246282 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="feast-init" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246290 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="feast-init" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246298 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="offline" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246305 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="offline" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246385 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="online" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246396 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="online" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246404 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="offline" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246411 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d485d8-bc58-4c29-b824-c263f4298c31" containerName="ui" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246419 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="offline" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246428 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="registry" Jul 16 17:43:12.247126 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.246437 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="791215b5-9831-46b1-a7eb-9d78cfa09b58" containerName="ui" Jul 16 17:43:12.250824 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.250808 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:43:12.253715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.253694 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 16 17:43:12.253853 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.253700 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jul 16 17:43:12.253853 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.253750 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 16 17:43:12.254802 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.254784 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-dsj9g\"" Jul 16 17:43:12.260166 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.260145 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-m8tc5"] Jul 16 17:43:12.334953 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.334915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/88e473fd-8ab9-442e-86ba-ca703d28154e-postgresdata\") pod \"postgres-864846d4f5-m8tc5\" (UID: \"88e473fd-8ab9-442e-86ba-ca703d28154e\") " pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:43:12.335129 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.334971 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzx5h\" (UniqueName: \"kubernetes.io/projected/88e473fd-8ab9-442e-86ba-ca703d28154e-kube-api-access-pzx5h\") pod \"postgres-864846d4f5-m8tc5\" (UID: \"88e473fd-8ab9-442e-86ba-ca703d28154e\") " pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:43:12.436517 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.436485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/88e473fd-8ab9-442e-86ba-ca703d28154e-postgresdata\") pod \"postgres-864846d4f5-m8tc5\" (UID: \"88e473fd-8ab9-442e-86ba-ca703d28154e\") " pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:43:12.436713 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.436543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzx5h\" (UniqueName: \"kubernetes.io/projected/88e473fd-8ab9-442e-86ba-ca703d28154e-kube-api-access-pzx5h\") pod \"postgres-864846d4f5-m8tc5\" (UID: \"88e473fd-8ab9-442e-86ba-ca703d28154e\") " pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:43:12.436891 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.436867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/88e473fd-8ab9-442e-86ba-ca703d28154e-postgresdata\") pod \"postgres-864846d4f5-m8tc5\" (UID: \"88e473fd-8ab9-442e-86ba-ca703d28154e\") " pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:43:12.445983 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.445961 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzx5h\" (UniqueName: \"kubernetes.io/projected/88e473fd-8ab9-442e-86ba-ca703d28154e-kube-api-access-pzx5h\") pod \"postgres-864846d4f5-m8tc5\" (UID: \"88e473fd-8ab9-442e-86ba-ca703d28154e\") " pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:43:12.566857 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.566809 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:43:12.694378 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.694346 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-m8tc5"] Jul 16 17:43:12.697779 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:43:12.697742 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88e473fd_8ab9_442e_86ba_ca703d28154e.slice/crio-121b31a799e5c213ebdb3cacacbb43d707c8a1f2d69aed7a949fa1b7e2a06e7f WatchSource:0}: Error finding container 121b31a799e5c213ebdb3cacacbb43d707c8a1f2d69aed7a949fa1b7e2a06e7f: Status 404 returned error can't find the container with id 121b31a799e5c213ebdb3cacacbb43d707c8a1f2d69aed7a949fa1b7e2a06e7f Jul 16 17:43:12.699669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:12.699652 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:43:13.235695 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:13.235659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-m8tc5" event={"ID":"88e473fd-8ab9-442e-86ba-ca703d28154e","Type":"ContainerStarted","Data":"121b31a799e5c213ebdb3cacacbb43d707c8a1f2d69aed7a949fa1b7e2a06e7f"} Jul 16 17:43:19.269402 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:19.269366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-m8tc5" event={"ID":"88e473fd-8ab9-442e-86ba-ca703d28154e","Type":"ContainerStarted","Data":"b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7"} Jul 16 17:43:19.286581 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:19.286535 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-m8tc5" podStartSLOduration=1.346248781 podStartE2EDuration="7.286521244s" totalImagesPullingTime="5.940272463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:19.286058616 +0000 UTC m=+779.957528832" watchObservedRunningTime="2026-07-16 17:43:19.286521244 +0000 UTC m=+779.957991448" Jul 16 17:43:22.903384 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:22.903349 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd"] Jul 16 17:43:22.909996 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:22.909972 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:22.914250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:22.914218 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jul 16 17:43:22.914569 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:22.914554 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-hdgvx\"" Jul 16 17:43:22.914655 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:22.914637 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jul 16 17:43:22.914914 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:22.914897 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 16 17:43:22.914979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:22.914935 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jul 16 17:43:22.927920 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:22.927889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd"] Jul 16 17:43:23.029223 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.029190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-registry-tls\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.029409 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.029242 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-offline-tls\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.029409 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.029268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/427bbc0b-fd16-48df-a927-a5da5612cf7c-feast-data\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.029409 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.029346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-online-tls\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.029409 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.029380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtzp8\" (UniqueName: \"kubernetes.io/projected/427bbc0b-fd16-48df-a927-a5da5612cf7c-kube-api-access-vtzp8\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.130582 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.130549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-offline-tls\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.130582 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.130584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/427bbc0b-fd16-48df-a927-a5da5612cf7c-feast-data\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.130825 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.130603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-online-tls\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.130825 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.130727 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtzp8\" (UniqueName: \"kubernetes.io/projected/427bbc0b-fd16-48df-a927-a5da5612cf7c-kube-api-access-vtzp8\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.131028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.130853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-registry-tls\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.131028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.130993 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/427bbc0b-fd16-48df-a927-a5da5612cf7c-feast-data\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.133148 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.133125 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-offline-tls\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.133262 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.133151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-registry-tls\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.133328 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.133306 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-online-tls\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.141009 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.140984 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtzp8\" (UniqueName: \"kubernetes.io/projected/427bbc0b-fd16-48df-a927-a5da5612cf7c-kube-api-access-vtzp8\") pod \"feast-credit-scoring-d74f49c7f-dz7hd\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.219311 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.219222 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:23.361346 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:23.361323 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd"] Jul 16 17:43:23.363246 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:43:23.363221 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427bbc0b_fd16_48df_a927_a5da5612cf7c.slice/crio-ff1aa3cc3a6d9adbf271d1b363793a6d72668ff301bd70986271c866707d4f6e WatchSource:0}: Error finding container ff1aa3cc3a6d9adbf271d1b363793a6d72668ff301bd70986271c866707d4f6e: Status 404 returned error can't find the container with id ff1aa3cc3a6d9adbf271d1b363793a6d72668ff301bd70986271c866707d4f6e Jul 16 17:43:24.294317 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:24.294278 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerStarted","Data":"d5662a342d92c4fb24d85ea078cc044751ae529038da9b08ff8ca552474cdb4a"} Jul 16 17:43:24.294799 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:24.294339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerStarted","Data":"ff1aa3cc3a6d9adbf271d1b363793a6d72668ff301bd70986271c866707d4f6e"} Jul 16 17:43:25.299764 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:25.299729 2565 generic.go:356] "Generic (PLEG): container finished" podID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerID="d5662a342d92c4fb24d85ea078cc044751ae529038da9b08ff8ca552474cdb4a" exitCode=0 Jul 16 17:43:25.300219 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:25.299820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerDied","Data":"d5662a342d92c4fb24d85ea078cc044751ae529038da9b08ff8ca552474cdb4a"} Jul 16 17:43:26.306008 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:26.305977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerStarted","Data":"dbc7293f2779c27b760b2440dd0312f2c78a6c4fd8106574590906b2ebb3c894"} Jul 16 17:43:32.334388 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:32.334350 2565 generic.go:356] "Generic (PLEG): container finished" podID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerID="dbc7293f2779c27b760b2440dd0312f2c78a6c4fd8106574590906b2ebb3c894" exitCode=0 Jul 16 17:43:32.334743 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:32.334420 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerDied","Data":"dbc7293f2779c27b760b2440dd0312f2c78a6c4fd8106574590906b2ebb3c894"} Jul 16 17:43:33.342607 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:33.342571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerStarted","Data":"a18ec6c3bdf3cc8f3794de6effe0e219b6901d86a5bf43660a739775f23cfb8a"} Jul 16 17:43:33.343118 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:33.342636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerStarted","Data":"aaad932b0ea27e348591a8805187b1e01c32a7a34a00a9998fcdb0e2e2759d3c"} Jul 16 17:43:33.343118 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:33.342662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerStarted","Data":"9d6273e9550f9d75890a9e2f29e5b80b2c25f064f5604ae28d976ebf55cde08b"} Jul 16 17:43:33.391494 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:33.391445 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podStartSLOduration=11.391428893 podStartE2EDuration="11.391428893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:33.389790871 +0000 UTC m=+794.061261077" watchObservedRunningTime="2026-07-16 17:43:33.391428893 +0000 UTC m=+794.062899097" Jul 16 17:43:35.220056 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:35.219999 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:35.220538 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:35.220092 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:35.220538 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:35.220118 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:35.221823 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:35.221785 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused" Jul 16 17:43:35.221979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:35.221817 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Jul 16 17:43:35.221979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:35.221864 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jul 16 17:43:38.221235 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:38.221183 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Jul 16 17:43:38.221762 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:38.221649 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:38.222101 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:38.222077 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:38.222668 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:38.222645 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:38.685824 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:38.685785 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:39.377565 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:39.377535 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:39.381054 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:39.381013 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:41.219650 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:41.219558 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Jul 16 17:43:43.220397 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:43.220359 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:44.220701 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:44.220672 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:43:44.221509 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:43:44.221488 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:44:47.709542 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:47.709507 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd"] Jul 16 17:44:47.710077 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:47.709745 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="registry" containerID="cri-o://9d6273e9550f9d75890a9e2f29e5b80b2c25f064f5604ae28d976ebf55cde08b" gracePeriod=30 Jul 16 17:44:47.710077 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:47.709812 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="offline" containerID="cri-o://a18ec6c3bdf3cc8f3794de6effe0e219b6901d86a5bf43660a739775f23cfb8a" gracePeriod=30 Jul 16 17:44:47.710077 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:47.709812 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="online" containerID="cri-o://aaad932b0ea27e348591a8805187b1e01c32a7a34a00a9998fcdb0e2e2759d3c" gracePeriod=30 Jul 16 17:44:47.729192 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:47.729163 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-m8tc5"] Jul 16 17:44:47.729356 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:47.729326 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-m8tc5" podUID="88e473fd-8ab9-442e-86ba-ca703d28154e" containerName="postgres" containerID="cri-o://b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7" gracePeriod=30 Jul 16 17:44:47.977953 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:47.977929 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:44:48.067079 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.067048 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/88e473fd-8ab9-442e-86ba-ca703d28154e-postgresdata\") pod \"88e473fd-8ab9-442e-86ba-ca703d28154e\" (UID: \"88e473fd-8ab9-442e-86ba-ca703d28154e\") " Jul 16 17:44:48.067226 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.067141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzx5h\" (UniqueName: \"kubernetes.io/projected/88e473fd-8ab9-442e-86ba-ca703d28154e-kube-api-access-pzx5h\") pod \"88e473fd-8ab9-442e-86ba-ca703d28154e\" (UID: \"88e473fd-8ab9-442e-86ba-ca703d28154e\") " Jul 16 17:44:48.069221 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.069190 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e473fd-8ab9-442e-86ba-ca703d28154e-kube-api-access-pzx5h" (OuterVolumeSpecName: "kube-api-access-pzx5h") pod "88e473fd-8ab9-442e-86ba-ca703d28154e" (UID: "88e473fd-8ab9-442e-86ba-ca703d28154e"). InnerVolumeSpecName "kube-api-access-pzx5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:44:48.096631 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.096596 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e473fd-8ab9-442e-86ba-ca703d28154e-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "88e473fd-8ab9-442e-86ba-ca703d28154e" (UID: "88e473fd-8ab9-442e-86ba-ca703d28154e"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:44:48.167919 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.167877 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzx5h\" (UniqueName: \"kubernetes.io/projected/88e473fd-8ab9-442e-86ba-ca703d28154e-kube-api-access-pzx5h\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:44:48.167919 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.167913 2565 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/88e473fd-8ab9-442e-86ba-ca703d28154e-postgresdata\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:44:48.691073 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.691039 2565 generic.go:356] "Generic (PLEG): container finished" podID="88e473fd-8ab9-442e-86ba-ca703d28154e" containerID="b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7" exitCode=0 Jul 16 17:44:48.691261 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.691084 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-m8tc5" Jul 16 17:44:48.691261 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.691124 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-m8tc5" event={"ID":"88e473fd-8ab9-442e-86ba-ca703d28154e","Type":"ContainerDied","Data":"b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7"} Jul 16 17:44:48.691261 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.691164 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-m8tc5" event={"ID":"88e473fd-8ab9-442e-86ba-ca703d28154e","Type":"ContainerDied","Data":"121b31a799e5c213ebdb3cacacbb43d707c8a1f2d69aed7a949fa1b7e2a06e7f"} Jul 16 17:44:48.691261 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.691181 2565 scope.go:117] "RemoveContainer" containerID="b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7" Jul 16 17:44:48.694382 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.694361 2565 generic.go:356] "Generic (PLEG): container finished" podID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerID="aaad932b0ea27e348591a8805187b1e01c32a7a34a00a9998fcdb0e2e2759d3c" exitCode=0 Jul 16 17:44:48.694531 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.694417 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerDied","Data":"aaad932b0ea27e348591a8805187b1e01c32a7a34a00a9998fcdb0e2e2759d3c"} Jul 16 17:44:48.700221 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.700205 2565 scope.go:117] "RemoveContainer" containerID="b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7" Jul 16 17:44:48.700488 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:44:48.700471 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7\": container with ID starting with b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7 not found: ID does not exist" containerID="b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7" Jul 16 17:44:48.700534 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.700496 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7"} err="failed to get container status \"b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7\": rpc error: code = NotFound desc = could not find container \"b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7\": container with ID starting with b20ad5c0134cbbab2c78133354b6b426abfa5d9be522b4bd359bd0ace9260cd7 not found: ID does not exist" Jul 16 17:44:48.712096 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.712075 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-m8tc5"] Jul 16 17:44:48.719156 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:48.719135 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-m8tc5"] Jul 16 17:44:49.377823 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:49.377785 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jul 16 17:44:49.876549 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:49.876510 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e473fd-8ab9-442e-86ba-ca703d28154e" path="/var/lib/kubelet/pods/88e473fd-8ab9-442e-86ba-ca703d28154e/volumes" Jul 16 17:44:59.378633 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:44:59.378582 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jul 16 17:45:09.377848 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:09.377801 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jul 16 17:45:09.378376 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:09.377989 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:45:17.822742 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:17.822710 2565 generic.go:356] "Generic (PLEG): container finished" podID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerID="a18ec6c3bdf3cc8f3794de6effe0e219b6901d86a5bf43660a739775f23cfb8a" exitCode=137 Jul 16 17:45:17.822742 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:17.822736 2565 generic.go:356] "Generic (PLEG): container finished" podID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerID="9d6273e9550f9d75890a9e2f29e5b80b2c25f064f5604ae28d976ebf55cde08b" exitCode=137 Jul 16 17:45:17.823136 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:17.822782 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerDied","Data":"a18ec6c3bdf3cc8f3794de6effe0e219b6901d86a5bf43660a739775f23cfb8a"} Jul 16 17:45:17.823136 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:17.822848 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerDied","Data":"9d6273e9550f9d75890a9e2f29e5b80b2c25f064f5604ae28d976ebf55cde08b"} Jul 16 17:45:18.222544 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.222503 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused" Jul 16 17:45:18.363226 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.363199 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:45:18.535740 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.535651 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-online-tls\") pod \"427bbc0b-fd16-48df-a927-a5da5612cf7c\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " Jul 16 17:45:18.535740 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.535706 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-offline-tls\") pod \"427bbc0b-fd16-48df-a927-a5da5612cf7c\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " Jul 16 17:45:18.535740 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.535735 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtzp8\" (UniqueName: \"kubernetes.io/projected/427bbc0b-fd16-48df-a927-a5da5612cf7c-kube-api-access-vtzp8\") pod \"427bbc0b-fd16-48df-a927-a5da5612cf7c\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " Jul 16 17:45:18.536065 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.535774 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-registry-tls\") pod \"427bbc0b-fd16-48df-a927-a5da5612cf7c\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " Jul 16 17:45:18.536065 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.535798 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/427bbc0b-fd16-48df-a927-a5da5612cf7c-feast-data\") pod \"427bbc0b-fd16-48df-a927-a5da5612cf7c\" (UID: \"427bbc0b-fd16-48df-a927-a5da5612cf7c\") " Jul 16 17:45:18.537925 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.537894 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-online-tls" (OuterVolumeSpecName: "online-tls") pod "427bbc0b-fd16-48df-a927-a5da5612cf7c" (UID: "427bbc0b-fd16-48df-a927-a5da5612cf7c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:45:18.538120 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.538094 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "427bbc0b-fd16-48df-a927-a5da5612cf7c" (UID: "427bbc0b-fd16-48df-a927-a5da5612cf7c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:45:18.538307 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.538286 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427bbc0b-fd16-48df-a927-a5da5612cf7c-kube-api-access-vtzp8" (OuterVolumeSpecName: "kube-api-access-vtzp8") pod "427bbc0b-fd16-48df-a927-a5da5612cf7c" (UID: "427bbc0b-fd16-48df-a927-a5da5612cf7c"). InnerVolumeSpecName "kube-api-access-vtzp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:45:18.538383 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.538365 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "427bbc0b-fd16-48df-a927-a5da5612cf7c" (UID: "427bbc0b-fd16-48df-a927-a5da5612cf7c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:45:18.544118 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.544091 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427bbc0b-fd16-48df-a927-a5da5612cf7c-feast-data" (OuterVolumeSpecName: "feast-data") pod "427bbc0b-fd16-48df-a927-a5da5612cf7c" (UID: "427bbc0b-fd16-48df-a927-a5da5612cf7c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:45:18.636439 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.636405 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:45:18.636439 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.636432 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:45:18.636439 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.636441 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtzp8\" (UniqueName: \"kubernetes.io/projected/427bbc0b-fd16-48df-a927-a5da5612cf7c-kube-api-access-vtzp8\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:45:18.636669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.636451 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/427bbc0b-fd16-48df-a927-a5da5612cf7c-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:45:18.636669 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.636459 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/427bbc0b-fd16-48df-a927-a5da5612cf7c-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:45:18.829790 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.829764 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" Jul 16 17:45:18.829790 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.829769 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd" event={"ID":"427bbc0b-fd16-48df-a927-a5da5612cf7c","Type":"ContainerDied","Data":"ff1aa3cc3a6d9adbf271d1b363793a6d72668ff301bd70986271c866707d4f6e"} Jul 16 17:45:18.830244 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.829832 2565 scope.go:117] "RemoveContainer" containerID="a18ec6c3bdf3cc8f3794de6effe0e219b6901d86a5bf43660a739775f23cfb8a" Jul 16 17:45:18.839028 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.839010 2565 scope.go:117] "RemoveContainer" containerID="aaad932b0ea27e348591a8805187b1e01c32a7a34a00a9998fcdb0e2e2759d3c" Jul 16 17:45:18.847095 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.847076 2565 scope.go:117] "RemoveContainer" containerID="9d6273e9550f9d75890a9e2f29e5b80b2c25f064f5604ae28d976ebf55cde08b" Jul 16 17:45:18.855546 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.855527 2565 scope.go:117] "RemoveContainer" containerID="dbc7293f2779c27b760b2440dd0312f2c78a6c4fd8106574590906b2ebb3c894" Jul 16 17:45:18.859480 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.859451 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd"] Jul 16 17:45:18.863402 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.863383 2565 scope.go:117] "RemoveContainer" containerID="d5662a342d92c4fb24d85ea078cc044751ae529038da9b08ff8ca552474cdb4a" Jul 16 17:45:18.885723 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:18.881801 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-d74f49c7f-dz7hd"] Jul 16 17:45:19.877123 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:45:19.877092 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" path="/var/lib/kubelet/pods/427bbc0b-fd16-48df-a927-a5da5612cf7c/volumes" Jul 16 17:47:20.646088 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:20.646052 2565 scope.go:117] "RemoveContainer" containerID="99234e7028de33f11eec9c580b573a75fdbd99aa2534235155b2f7726a4aec90" Jul 16 17:47:20.654364 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:20.654342 2565 scope.go:117] "RemoveContainer" containerID="cb2d347573748ae235399890007ff685bc4784701491428e4fe0d6a49ae06b7d" Jul 16 17:47:21.523680 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.523648 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5nf24"] Jul 16 17:47:21.524048 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524036 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="registry" Jul 16 17:47:21.524096 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524053 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="registry" Jul 16 17:47:21.524096 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524068 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="feast-apply" Jul 16 17:47:21.524096 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524074 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="feast-apply" Jul 16 17:47:21.524096 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524082 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="offline" Jul 16 17:47:21.524096 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524087 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="offline" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524099 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e473fd-8ab9-442e-86ba-ca703d28154e" containerName="postgres" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524104 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e473fd-8ab9-442e-86ba-ca703d28154e" containerName="postgres" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524120 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="feast-init" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524125 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="feast-init" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524133 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="online" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524142 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="online" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524196 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="online" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524204 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="registry" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524210 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="427bbc0b-fd16-48df-a927-a5da5612cf7c" containerName="offline" Jul 16 17:47:21.524259 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.524222 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e473fd-8ab9-442e-86ba-ca703d28154e" containerName="postgres" Jul 16 17:47:21.528689 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.528667 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" Jul 16 17:47:21.531941 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.531921 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jul 16 17:47:21.532917 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.532897 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-fkcn8\"" Jul 16 17:47:21.533039 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.532935 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jul 16 17:47:21.549434 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.549407 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5nf24"] Jul 16 17:47:21.639496 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.639458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxnt6\" (UniqueName: \"kubernetes.io/projected/20f3e31b-1d28-4adf-b928-bcd5b5f14412-kube-api-access-fxnt6\") pod \"redis-69bbd798b6-5nf24\" (UID: \"20f3e31b-1d28-4adf-b928-bcd5b5f14412\") " pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" Jul 16 17:47:21.740492 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.740454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxnt6\" (UniqueName: \"kubernetes.io/projected/20f3e31b-1d28-4adf-b928-bcd5b5f14412-kube-api-access-fxnt6\") pod \"redis-69bbd798b6-5nf24\" (UID: \"20f3e31b-1d28-4adf-b928-bcd5b5f14412\") " pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" Jul 16 17:47:21.758814 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.758780 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxnt6\" (UniqueName: \"kubernetes.io/projected/20f3e31b-1d28-4adf-b928-bcd5b5f14412-kube-api-access-fxnt6\") pod \"redis-69bbd798b6-5nf24\" (UID: \"20f3e31b-1d28-4adf-b928-bcd5b5f14412\") " pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" Jul 16 17:47:21.854427 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.854396 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" Jul 16 17:47:21.988788 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:21.988765 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5nf24"] Jul 16 17:47:21.991176 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:47:21.991142 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20f3e31b_1d28_4adf_b928_bcd5b5f14412.slice/crio-397686e9de0b29d0fddb9acc33ca46a07dbed6ae6216786be642bd2396d2c71d WatchSource:0}: Error finding container 397686e9de0b29d0fddb9acc33ca46a07dbed6ae6216786be642bd2396d2c71d: Status 404 returned error can't find the container with id 397686e9de0b29d0fddb9acc33ca46a07dbed6ae6216786be642bd2396d2c71d Jul 16 17:47:22.327131 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:22.327099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" event={"ID":"20f3e31b-1d28-4adf-b928-bcd5b5f14412","Type":"ContainerStarted","Data":"397686e9de0b29d0fddb9acc33ca46a07dbed6ae6216786be642bd2396d2c71d"} Jul 16 17:47:24.337131 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:24.337091 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" event={"ID":"20f3e31b-1d28-4adf-b928-bcd5b5f14412","Type":"ContainerStarted","Data":"a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6"} Jul 16 17:47:24.360278 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:24.360226 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" podStartSLOduration=1.98059691 podStartE2EDuration="3.36021292s" totalImagesPullingTime="1.37961601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:47:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:47:24.358996936 +0000 UTC m=+1025.030467140" watchObservedRunningTime="2026-07-16 17:47:24.36021292 +0000 UTC m=+1025.031683123" Jul 16 17:47:26.997898 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:26.997859 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kr5t9"] Jul 16 17:47:27.001375 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.001354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:47:27.004443 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.004422 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jul 16 17:47:27.012925 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.012901 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kr5t9"] Jul 16 17:47:27.086036 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.085998 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5qx\" (UniqueName: \"kubernetes.io/projected/659b7892-52af-4545-8f67-ca6cc4296750-kube-api-access-dg5qx\") pod \"postgres-864846d4f5-kr5t9\" (UID: \"659b7892-52af-4545-8f67-ca6cc4296750\") " pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:47:27.086180 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.086138 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/659b7892-52af-4545-8f67-ca6cc4296750-postgresdata\") pod \"postgres-864846d4f5-kr5t9\" (UID: \"659b7892-52af-4545-8f67-ca6cc4296750\") " pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:47:27.186672 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.186633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/659b7892-52af-4545-8f67-ca6cc4296750-postgresdata\") pod \"postgres-864846d4f5-kr5t9\" (UID: \"659b7892-52af-4545-8f67-ca6cc4296750\") " pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:47:27.186810 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.186678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5qx\" (UniqueName: \"kubernetes.io/projected/659b7892-52af-4545-8f67-ca6cc4296750-kube-api-access-dg5qx\") pod \"postgres-864846d4f5-kr5t9\" (UID: \"659b7892-52af-4545-8f67-ca6cc4296750\") " pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:47:27.187031 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.187010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/659b7892-52af-4545-8f67-ca6cc4296750-postgresdata\") pod \"postgres-864846d4f5-kr5t9\" (UID: \"659b7892-52af-4545-8f67-ca6cc4296750\") " pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:47:27.196674 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.196645 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5qx\" (UniqueName: \"kubernetes.io/projected/659b7892-52af-4545-8f67-ca6cc4296750-kube-api-access-dg5qx\") pod \"postgres-864846d4f5-kr5t9\" (UID: \"659b7892-52af-4545-8f67-ca6cc4296750\") " pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:47:27.309507 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.309468 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:47:27.441743 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:27.441709 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kr5t9"] Jul 16 17:47:27.444593 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:47:27.444568 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod659b7892_52af_4545_8f67_ca6cc4296750.slice/crio-9cde553f17425bc51fa196b98edbf013fbc615f2757212ccae54d6d56c162d73 WatchSource:0}: Error finding container 9cde553f17425bc51fa196b98edbf013fbc615f2757212ccae54d6d56c162d73: Status 404 returned error can't find the container with id 9cde553f17425bc51fa196b98edbf013fbc615f2757212ccae54d6d56c162d73 Jul 16 17:47:28.356966 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:28.356927 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" event={"ID":"659b7892-52af-4545-8f67-ca6cc4296750","Type":"ContainerStarted","Data":"253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37"} Jul 16 17:47:28.357429 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:28.356973 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" event={"ID":"659b7892-52af-4545-8f67-ca6cc4296750","Type":"ContainerStarted","Data":"9cde553f17425bc51fa196b98edbf013fbc615f2757212ccae54d6d56c162d73"} Jul 16 17:47:32.593807 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.593731 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" podStartSLOduration=6.593707524 podStartE2EDuration="6.593707524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:47:28.390159207 +0000 UTC m=+1029.061629411" watchObservedRunningTime="2026-07-16 17:47:32.593707524 +0000 UTC m=+1033.265177725" Jul 16 17:47:32.594775 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.594756 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6"] Jul 16 17:47:32.598413 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.598398 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.608322 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.608304 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jul 16 17:47:32.608440 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.608363 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-w6xk6\"" Jul 16 17:47:32.608440 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.608391 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jul 16 17:47:32.608440 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.608393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jul 16 17:47:32.652718 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.652689 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6"] Jul 16 17:47:32.735807 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.735775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-online-tls\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.736001 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.735892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-feast-data\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.736001 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.735931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgr5z\" (UniqueName: \"kubernetes.io/projected/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-kube-api-access-lgr5z\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.736104 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.736004 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-registry-tls\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.837437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.837394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-feast-data\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.837437 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.837447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgr5z\" (UniqueName: \"kubernetes.io/projected/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-kube-api-access-lgr5z\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.837711 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.837563 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-registry-tls\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.837711 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.837618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-online-tls\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.837821 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:47:32.837708 2565 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jul 16 17:47:32.837821 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:47:32.837797 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-registry-tls podName:0e67c3b8-e9aa-403e-b6d9-dff5c5db848b nodeName:}" failed. No retries permitted until 2026-07-16 17:47:33.337775825 +0000 UTC m=+1034.009246007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-registry-tls") pod "feast-credit-scoring-569dccd749-lgtr6" (UID: "0e67c3b8-e9aa-403e-b6d9-dff5c5db848b") : secret "feast-credit-scoring-registry-tls" not found Jul 16 17:47:32.837977 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.837875 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-feast-data\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.840154 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.840136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-online-tls\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:32.848909 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:32.848826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgr5z\" (UniqueName: \"kubernetes.io/projected/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-kube-api-access-lgr5z\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:33.342523 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:33.342489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-registry-tls\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:33.344963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:33.344933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-registry-tls\") pod \"feast-credit-scoring-569dccd749-lgtr6\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:33.507463 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:33.507429 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:33.651687 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:33.651640 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6"] Jul 16 17:47:33.653997 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:47:33.653971 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e67c3b8_e9aa_403e_b6d9_dff5c5db848b.slice/crio-a849e8bb773c5f292cf7236ba17e922204853a37a3a9209540852d396274e9f5 WatchSource:0}: Error finding container a849e8bb773c5f292cf7236ba17e922204853a37a3a9209540852d396274e9f5: Status 404 returned error can't find the container with id a849e8bb773c5f292cf7236ba17e922204853a37a3a9209540852d396274e9f5 Jul 16 17:47:34.382278 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:34.382240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerStarted","Data":"121dc4b70e52f575f13c5bbf84753b84b442d17f13d63f2f7db72a8d52480314"} Jul 16 17:47:34.382450 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:34.382302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerStarted","Data":"a849e8bb773c5f292cf7236ba17e922204853a37a3a9209540852d396274e9f5"} Jul 16 17:47:35.387603 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:35.387523 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerID="121dc4b70e52f575f13c5bbf84753b84b442d17f13d63f2f7db72a8d52480314" exitCode=0 Jul 16 17:47:35.387976 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:35.387612 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerDied","Data":"121dc4b70e52f575f13c5bbf84753b84b442d17f13d63f2f7db72a8d52480314"} Jul 16 17:47:36.394269 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:36.394233 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerStarted","Data":"236bbd68cef1c5e6373e460efbc03bf6503d866d996a0316be78166fad7497c6"} Jul 16 17:47:42.424136 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:42.424106 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerID="236bbd68cef1c5e6373e460efbc03bf6503d866d996a0316be78166fad7497c6" exitCode=0 Jul 16 17:47:42.424532 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:42.424184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerDied","Data":"236bbd68cef1c5e6373e460efbc03bf6503d866d996a0316be78166fad7497c6"} Jul 16 17:47:43.432818 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:43.432786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerStarted","Data":"c6d3ab44e9bf4a0ff3c3b304f5d85d56de5ed51d9e7ceda8d3e5aac00b69284b"} Jul 16 17:47:43.433210 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:43.432853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerStarted","Data":"fe3a4f663fc25912c408d238738823a4334125a6f4d3acc39d99643e850e06bc"} Jul 16 17:47:43.461944 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:43.461866 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podStartSLOduration=11.461815789 podStartE2EDuration="11.461815789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:47:43.459961565 +0000 UTC m=+1044.131431769" watchObservedRunningTime="2026-07-16 17:47:43.461815789 +0000 UTC m=+1044.133285994" Jul 16 17:47:43.507801 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:43.507773 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:43.507969 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:43.507911 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:45.508366 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:45.508324 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:45.508829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:45.508405 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:45.510050 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:45.510021 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jul 16 17:47:45.510217 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:45.510043 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Jul 16 17:47:48.508644 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:48.508600 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Jul 16 17:47:48.685975 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:48.685941 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:48.689317 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:48.689287 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:51.507831 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:51.507781 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Jul 16 17:47:54.509070 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:54.509041 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:54.509831 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:54.509811 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:47:58.679412 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.679375 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8"] Jul 16 17:47:58.685920 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.685900 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.694434 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.694407 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-vhx69\"" Jul 16 17:47:58.694575 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.694419 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jul 16 17:47:58.694575 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.694419 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 16 17:47:58.694703 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.694629 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jul 16 17:47:58.709635 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.709611 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8"] Jul 16 17:47:58.763184 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.763144 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-registry-tls\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.763184 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.763186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mcr\" (UniqueName: \"kubernetes.io/projected/d128ee21-8e4e-498a-8ce6-daf49693054d-kube-api-access-c5mcr\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.763399 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.763274 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d128ee21-8e4e-498a-8ce6-daf49693054d-feast-data\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.763399 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.763317 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-online-tls\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.864251 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.864213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d128ee21-8e4e-498a-8ce6-daf49693054d-feast-data\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.864428 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.864260 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-online-tls\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.864428 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.864335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-registry-tls\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.864428 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.864376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mcr\" (UniqueName: \"kubernetes.io/projected/d128ee21-8e4e-498a-8ce6-daf49693054d-kube-api-access-c5mcr\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.864678 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.864658 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d128ee21-8e4e-498a-8ce6-daf49693054d-feast-data\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.866974 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.866946 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-online-tls\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.867085 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.867071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-registry-tls\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.877648 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.877617 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mcr\" (UniqueName: \"kubernetes.io/projected/d128ee21-8e4e-498a-8ce6-daf49693054d-kube-api-access-c5mcr\") pod \"feast-driver-ranking-6756b54b-fwrx8\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:58.995662 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:58.995560 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:47:59.147250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:59.147219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8"] Jul 16 17:47:59.149397 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:47:59.149369 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd128ee21_8e4e_498a_8ce6_daf49693054d.slice/crio-400af4b0f87990dc711daea8a1805203efe2a0205e8fd2d7f198b628924d7246 WatchSource:0}: Error finding container 400af4b0f87990dc711daea8a1805203efe2a0205e8fd2d7f198b628924d7246: Status 404 returned error can't find the container with id 400af4b0f87990dc711daea8a1805203efe2a0205e8fd2d7f198b628924d7246 Jul 16 17:47:59.523971 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:59.523924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerStarted","Data":"0e030c60ca24c03f61289bd9163f1066489446eecc075f401d37faee4a8996f5"} Jul 16 17:47:59.524165 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:47:59.523998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerStarted","Data":"400af4b0f87990dc711daea8a1805203efe2a0205e8fd2d7f198b628924d7246"} Jul 16 17:48:04.549417 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:04.549378 2565 generic.go:356] "Generic (PLEG): container finished" podID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerID="0e030c60ca24c03f61289bd9163f1066489446eecc075f401d37faee4a8996f5" exitCode=0 Jul 16 17:48:04.549827 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:04.549449 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerDied","Data":"0e030c60ca24c03f61289bd9163f1066489446eecc075f401d37faee4a8996f5"} Jul 16 17:48:05.558150 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:05.558116 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerStarted","Data":"cff41bdc7effadc4580fcd266fc36707aab9eaf76b50ae27d5012b5d86a8a037"} Jul 16 17:48:11.591318 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:11.591283 2565 generic.go:356] "Generic (PLEG): container finished" podID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerID="cff41bdc7effadc4580fcd266fc36707aab9eaf76b50ae27d5012b5d86a8a037" exitCode=0 Jul 16 17:48:11.591689 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:11.591351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerDied","Data":"cff41bdc7effadc4580fcd266fc36707aab9eaf76b50ae27d5012b5d86a8a037"} Jul 16 17:48:12.598960 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:12.598922 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerStarted","Data":"f02c1ade4676ee6ff34be9a611987b41f8c2a77490cc7f9f0ce7499f5b58ad84"} Jul 16 17:48:12.599344 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:12.598997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerStarted","Data":"d3af612f94b6007d75d265d88700d103ad250c67f49efa38014650b0e5dba59e"} Jul 16 17:48:12.628700 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:12.628639 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podStartSLOduration=14.628622063 podStartE2EDuration="14.628622063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:12.626930012 +0000 UTC m=+1073.298400212" watchObservedRunningTime="2026-07-16 17:48:12.628622063 +0000 UTC m=+1073.300092295" Jul 16 17:48:13.996630 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:13.996596 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:48:13.997074 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:13.996670 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:48:13.998352 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:13.998313 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Jul 16 17:48:13.998504 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:13.998352 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jul 16 17:48:16.996926 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:16.996885 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Jul 16 17:48:16.996926 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:16.996902 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jul 16 17:48:18.995895 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:18.995833 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:48:18.996384 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:18.995915 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:48:19.996855 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:19.996783 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Jul 16 17:48:20.002488 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:20.002456 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:48:20.005977 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:20.005950 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:48:20.668103 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:20.668074 2565 scope.go:117] "RemoveContainer" containerID="bb1694a69b576c966676b9fef12f4964783fa4886a387536b320562fbd1dd057" Jul 16 17:48:20.679237 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:20.679216 2565 scope.go:117] "RemoveContainer" containerID="43c4f5ff4300bb8d875a0e7e2808706e93da421175482ac6c481c2f569aa4831" Jul 16 17:48:20.688817 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:20.688799 2565 scope.go:117] "RemoveContainer" containerID="9bb8def803901e2405b0aae824448fa8e21253ae44953107e1b32616ef0184d5" Jul 16 17:48:22.996936 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:22.996906 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:48:22.997686 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:48:22.997670 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:49:38.097472 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.097440 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l"] Jul 16 17:49:38.098017 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.097708 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" podUID="341402da-904c-44a4-b227-dcc85952c9cc" containerName="manager" containerID="cri-o://13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c" gracePeriod=10 Jul 16 17:49:38.354152 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.354079 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" Jul 16 17:49:38.433462 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.433430 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jzgq\" (UniqueName: \"kubernetes.io/projected/341402da-904c-44a4-b227-dcc85952c9cc-kube-api-access-2jzgq\") pod \"341402da-904c-44a4-b227-dcc85952c9cc\" (UID: \"341402da-904c-44a4-b227-dcc85952c9cc\") " Jul 16 17:49:38.436802 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.436763 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341402da-904c-44a4-b227-dcc85952c9cc-kube-api-access-2jzgq" (OuterVolumeSpecName: "kube-api-access-2jzgq") pod "341402da-904c-44a4-b227-dcc85952c9cc" (UID: "341402da-904c-44a4-b227-dcc85952c9cc"). InnerVolumeSpecName "kube-api-access-2jzgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:49:38.466973 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.466939 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6"] Jul 16 17:49:38.467189 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.467160 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="registry" containerID="cri-o://fe3a4f663fc25912c408d238738823a4334125a6f4d3acc39d99643e850e06bc" gracePeriod=30 Jul 16 17:49:38.467257 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.467214 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="online" containerID="cri-o://c6d3ab44e9bf4a0ff3c3b304f5d85d56de5ed51d9e7ceda8d3e5aac00b69284b" gracePeriod=30 Jul 16 17:49:38.476536 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.476505 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8"] Jul 16 17:49:38.476757 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.476733 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="registry" containerID="cri-o://d3af612f94b6007d75d265d88700d103ad250c67f49efa38014650b0e5dba59e" gracePeriod=30 Jul 16 17:49:38.476832 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.476768 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="online" containerID="cri-o://f02c1ade4676ee6ff34be9a611987b41f8c2a77490cc7f9f0ce7499f5b58ad84" gracePeriod=30 Jul 16 17:49:38.534872 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.534805 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jzgq\" (UniqueName: \"kubernetes.io/projected/341402da-904c-44a4-b227-dcc85952c9cc-kube-api-access-2jzgq\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:49:38.687103 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:38.687015 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jul 16 17:49:39.012037 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.011946 2565 generic.go:356] "Generic (PLEG): container finished" podID="341402da-904c-44a4-b227-dcc85952c9cc" containerID="13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c" exitCode=0 Jul 16 17:49:39.012037 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.011995 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" Jul 16 17:49:39.012283 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.012035 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" event={"ID":"341402da-904c-44a4-b227-dcc85952c9cc","Type":"ContainerDied","Data":"13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c"} Jul 16 17:49:39.012283 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.012094 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l" event={"ID":"341402da-904c-44a4-b227-dcc85952c9cc","Type":"ContainerDied","Data":"ce45e513c4ca77aca4a6417b5b9223c734af535a05b2ea888c698d160e0297e9"} Jul 16 17:49:39.012283 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.012121 2565 scope.go:117] "RemoveContainer" containerID="13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c" Jul 16 17:49:39.026368 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.026055 2565 scope.go:117] "RemoveContainer" containerID="13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c" Jul 16 17:49:39.026488 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:49:39.026432 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c\": container with ID starting with 13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c not found: ID does not exist" containerID="13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c" Jul 16 17:49:39.026488 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.026453 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c"} err="failed to get container status \"13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c\": rpc error: code = NotFound desc = could not find container \"13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c\": container with ID starting with 13e224d149eed1677bce3e61451cbe22c4813f3b1b22ef95e07e725dd7c2365c not found: ID does not exist" Jul 16 17:49:39.045955 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.043471 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l"] Jul 16 17:49:39.059300 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.059195 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5bbfd9c774-pd76l"] Jul 16 17:49:39.876468 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:39.876424 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="341402da-904c-44a4-b227-dcc85952c9cc" path="/var/lib/kubelet/pods/341402da-904c-44a4-b227-dcc85952c9cc/volumes" Jul 16 17:49:40.003765 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:40.003726 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jul 16 17:49:40.019727 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:40.019697 2565 generic.go:356] "Generic (PLEG): container finished" podID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerID="f02c1ade4676ee6ff34be9a611987b41f8c2a77490cc7f9f0ce7499f5b58ad84" exitCode=0 Jul 16 17:49:40.019899 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:40.019771 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerDied","Data":"f02c1ade4676ee6ff34be9a611987b41f8c2a77490cc7f9f0ce7499f5b58ad84"} Jul 16 17:49:40.021889 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:40.021870 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerID="c6d3ab44e9bf4a0ff3c3b304f5d85d56de5ed51d9e7ceda8d3e5aac00b69284b" exitCode=0 Jul 16 17:49:40.021986 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:40.021906 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerDied","Data":"c6d3ab44e9bf4a0ff3c3b304f5d85d56de5ed51d9e7ceda8d3e5aac00b69284b"} Jul 16 17:49:40.724149 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:40.724072 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kr5t9"] Jul 16 17:49:40.724304 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:40.724249 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" podUID="659b7892-52af-4545-8f67-ca6cc4296750" containerName="postgres" containerID="cri-o://253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37" gracePeriod=30 Jul 16 17:49:40.737292 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:40.737264 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5nf24"] Jul 16 17:49:40.737444 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:40.737427 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" podUID="20f3e31b-1d28-4adf-b928-bcd5b5f14412" containerName="redis" containerID="cri-o://a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6" gracePeriod=30 Jul 16 17:49:41.006903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.006879 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:49:41.009907 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.009882 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" Jul 16 17:49:41.028944 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.028910 2565 generic.go:356] "Generic (PLEG): container finished" podID="659b7892-52af-4545-8f67-ca6cc4296750" containerID="253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37" exitCode=0 Jul 16 17:49:41.029095 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.028983 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" Jul 16 17:49:41.029095 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.029053 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" event={"ID":"659b7892-52af-4545-8f67-ca6cc4296750","Type":"ContainerDied","Data":"253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37"} Jul 16 17:49:41.029095 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.029088 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kr5t9" event={"ID":"659b7892-52af-4545-8f67-ca6cc4296750","Type":"ContainerDied","Data":"9cde553f17425bc51fa196b98edbf013fbc615f2757212ccae54d6d56c162d73"} Jul 16 17:49:41.029267 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.029109 2565 scope.go:117] "RemoveContainer" containerID="253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37" Jul 16 17:49:41.031657 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.031629 2565 generic.go:356] "Generic (PLEG): container finished" podID="20f3e31b-1d28-4adf-b928-bcd5b5f14412" containerID="a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6" exitCode=0 Jul 16 17:49:41.031785 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.031671 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" event={"ID":"20f3e31b-1d28-4adf-b928-bcd5b5f14412","Type":"ContainerDied","Data":"a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6"} Jul 16 17:49:41.031785 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.031678 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" Jul 16 17:49:41.031785 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.031702 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-5nf24" event={"ID":"20f3e31b-1d28-4adf-b928-bcd5b5f14412","Type":"ContainerDied","Data":"397686e9de0b29d0fddb9acc33ca46a07dbed6ae6216786be642bd2396d2c71d"} Jul 16 17:49:41.038951 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.038933 2565 scope.go:117] "RemoveContainer" containerID="253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37" Jul 16 17:49:41.039273 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:49:41.039233 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37\": container with ID starting with 253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37 not found: ID does not exist" containerID="253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37" Jul 16 17:49:41.039370 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.039286 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37"} err="failed to get container status \"253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37\": rpc error: code = NotFound desc = could not find container \"253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37\": container with ID starting with 253e0326b7ac6e90b70c34a93ca7b02a75acb4d2ad1630ae1a63aede6c95af37 not found: ID does not exist" Jul 16 17:49:41.039370 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.039313 2565 scope.go:117] "RemoveContainer" containerID="a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6" Jul 16 17:49:41.048713 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.048577 2565 scope.go:117] "RemoveContainer" containerID="a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6" Jul 16 17:49:41.048998 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:49:41.048964 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6\": container with ID starting with a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6 not found: ID does not exist" containerID="a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6" Jul 16 17:49:41.049143 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.049018 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6"} err="failed to get container status \"a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6\": rpc error: code = NotFound desc = could not find container \"a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6\": container with ID starting with a896959d6afd580d5970539fe4d3472eefdd6c407bc346b7aaf4569331dd72f6 not found: ID does not exist" Jul 16 17:49:41.059230 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.059211 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxnt6\" (UniqueName: \"kubernetes.io/projected/20f3e31b-1d28-4adf-b928-bcd5b5f14412-kube-api-access-fxnt6\") pod \"20f3e31b-1d28-4adf-b928-bcd5b5f14412\" (UID: \"20f3e31b-1d28-4adf-b928-bcd5b5f14412\") " Jul 16 17:49:41.059356 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.059325 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/659b7892-52af-4545-8f67-ca6cc4296750-postgresdata\") pod \"659b7892-52af-4545-8f67-ca6cc4296750\" (UID: \"659b7892-52af-4545-8f67-ca6cc4296750\") " Jul 16 17:49:41.059424 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.059383 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg5qx\" (UniqueName: \"kubernetes.io/projected/659b7892-52af-4545-8f67-ca6cc4296750-kube-api-access-dg5qx\") pod \"659b7892-52af-4545-8f67-ca6cc4296750\" (UID: \"659b7892-52af-4545-8f67-ca6cc4296750\") " Jul 16 17:49:41.061493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.061469 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f3e31b-1d28-4adf-b928-bcd5b5f14412-kube-api-access-fxnt6" (OuterVolumeSpecName: "kube-api-access-fxnt6") pod "20f3e31b-1d28-4adf-b928-bcd5b5f14412" (UID: "20f3e31b-1d28-4adf-b928-bcd5b5f14412"). InnerVolumeSpecName "kube-api-access-fxnt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:49:41.061622 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.061567 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659b7892-52af-4545-8f67-ca6cc4296750-kube-api-access-dg5qx" (OuterVolumeSpecName: "kube-api-access-dg5qx") pod "659b7892-52af-4545-8f67-ca6cc4296750" (UID: "659b7892-52af-4545-8f67-ca6cc4296750"). InnerVolumeSpecName "kube-api-access-dg5qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:49:41.091274 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.091239 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659b7892-52af-4545-8f67-ca6cc4296750-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "659b7892-52af-4545-8f67-ca6cc4296750" (UID: "659b7892-52af-4545-8f67-ca6cc4296750"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:49:41.160615 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.160564 2565 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/659b7892-52af-4545-8f67-ca6cc4296750-postgresdata\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:49:41.160615 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.160614 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg5qx\" (UniqueName: \"kubernetes.io/projected/659b7892-52af-4545-8f67-ca6cc4296750-kube-api-access-dg5qx\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:49:41.160615 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.160623 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxnt6\" (UniqueName: \"kubernetes.io/projected/20f3e31b-1d28-4adf-b928-bcd5b5f14412-kube-api-access-fxnt6\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:49:41.357457 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.357432 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kr5t9"] Jul 16 17:49:41.364763 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.364736 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kr5t9"] Jul 16 17:49:41.379470 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.379444 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5nf24"] Jul 16 17:49:41.381712 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.381692 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5nf24"] Jul 16 17:49:41.876834 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.876802 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20f3e31b-1d28-4adf-b928-bcd5b5f14412" path="/var/lib/kubelet/pods/20f3e31b-1d28-4adf-b928-bcd5b5f14412/volumes" Jul 16 17:49:41.877166 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:41.877153 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="659b7892-52af-4545-8f67-ca6cc4296750" path="/var/lib/kubelet/pods/659b7892-52af-4545-8f67-ca6cc4296750/volumes" Jul 16 17:49:48.686737 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:48.686693 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jul 16 17:49:50.003276 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:50.003232 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jul 16 17:49:58.687142 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:58.687096 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jul 16 17:49:58.687596 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:49:58.687233 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:50:00.003429 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:00.003388 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jul 16 17:50:00.003778 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:00.003516 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:50:08.687295 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:08.687252 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jul 16 17:50:09.157694 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.157661 2565 generic.go:356] "Generic (PLEG): container finished" podID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerID="d3af612f94b6007d75d265d88700d103ad250c67f49efa38014650b0e5dba59e" exitCode=137 Jul 16 17:50:09.157829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.157697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerDied","Data":"d3af612f94b6007d75d265d88700d103ad250c67f49efa38014650b0e5dba59e"} Jul 16 17:50:09.157829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.157775 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" event={"ID":"d128ee21-8e4e-498a-8ce6-daf49693054d","Type":"ContainerDied","Data":"400af4b0f87990dc711daea8a1805203efe2a0205e8fd2d7f198b628924d7246"} Jul 16 17:50:09.157829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.157801 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="400af4b0f87990dc711daea8a1805203efe2a0205e8fd2d7f198b628924d7246" Jul 16 17:50:09.160111 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.160083 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerID="fe3a4f663fc25912c408d238738823a4334125a6f4d3acc39d99643e850e06bc" exitCode=137 Jul 16 17:50:09.160245 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.160156 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerDied","Data":"fe3a4f663fc25912c408d238738823a4334125a6f4d3acc39d99643e850e06bc"} Jul 16 17:50:09.160245 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.160220 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" event={"ID":"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b","Type":"ContainerDied","Data":"a849e8bb773c5f292cf7236ba17e922204853a37a3a9209540852d396274e9f5"} Jul 16 17:50:09.160245 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.160240 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a849e8bb773c5f292cf7236ba17e922204853a37a3a9209540852d396274e9f5" Jul 16 17:50:09.163993 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.163970 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:50:09.168871 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.168852 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:50:09.305400 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.305365 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-registry-tls\") pod \"d128ee21-8e4e-498a-8ce6-daf49693054d\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " Jul 16 17:50:09.305576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.305417 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-feast-data\") pod \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " Jul 16 17:50:09.305576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.305451 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-registry-tls\") pod \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " Jul 16 17:50:09.305576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.305471 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5mcr\" (UniqueName: \"kubernetes.io/projected/d128ee21-8e4e-498a-8ce6-daf49693054d-kube-api-access-c5mcr\") pod \"d128ee21-8e4e-498a-8ce6-daf49693054d\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " Jul 16 17:50:09.305576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.305494 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-online-tls\") pod \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " Jul 16 17:50:09.305576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.305532 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgr5z\" (UniqueName: \"kubernetes.io/projected/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-kube-api-access-lgr5z\") pod \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\" (UID: \"0e67c3b8-e9aa-403e-b6d9-dff5c5db848b\") " Jul 16 17:50:09.305576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.305551 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d128ee21-8e4e-498a-8ce6-daf49693054d-feast-data\") pod \"d128ee21-8e4e-498a-8ce6-daf49693054d\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " Jul 16 17:50:09.305918 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.305614 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-online-tls\") pod \"d128ee21-8e4e-498a-8ce6-daf49693054d\" (UID: \"d128ee21-8e4e-498a-8ce6-daf49693054d\") " Jul 16 17:50:09.306228 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.306157 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d128ee21-8e4e-498a-8ce6-daf49693054d-feast-data" (OuterVolumeSpecName: "feast-data") pod "d128ee21-8e4e-498a-8ce6-daf49693054d" (UID: "d128ee21-8e4e-498a-8ce6-daf49693054d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:09.308416 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.308389 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-online-tls" (OuterVolumeSpecName: "online-tls") pod "0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" (UID: "0e67c3b8-e9aa-403e-b6d9-dff5c5db848b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:50:09.308535 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.308451 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d128ee21-8e4e-498a-8ce6-daf49693054d-kube-api-access-c5mcr" (OuterVolumeSpecName: "kube-api-access-c5mcr") pod "d128ee21-8e4e-498a-8ce6-daf49693054d" (UID: "d128ee21-8e4e-498a-8ce6-daf49693054d"). InnerVolumeSpecName "kube-api-access-c5mcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:50:09.308535 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.308497 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-online-tls" (OuterVolumeSpecName: "online-tls") pod "d128ee21-8e4e-498a-8ce6-daf49693054d" (UID: "d128ee21-8e4e-498a-8ce6-daf49693054d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:50:09.308619 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.308523 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-kube-api-access-lgr5z" (OuterVolumeSpecName: "kube-api-access-lgr5z") pod "0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" (UID: "0e67c3b8-e9aa-403e-b6d9-dff5c5db848b"). InnerVolumeSpecName "kube-api-access-lgr5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:50:09.308619 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.308567 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d128ee21-8e4e-498a-8ce6-daf49693054d" (UID: "d128ee21-8e4e-498a-8ce6-daf49693054d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:50:09.308619 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.308566 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" (UID: "0e67c3b8-e9aa-403e-b6d9-dff5c5db848b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:50:09.315047 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.315022 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-feast-data" (OuterVolumeSpecName: "feast-data") pod "0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" (UID: "0e67c3b8-e9aa-403e-b6d9-dff5c5db848b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:09.406755 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.406722 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:50:09.406755 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.406751 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5mcr\" (UniqueName: \"kubernetes.io/projected/d128ee21-8e4e-498a-8ce6-daf49693054d-kube-api-access-c5mcr\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:50:09.406963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.406779 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:50:09.406963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.406788 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgr5z\" (UniqueName: \"kubernetes.io/projected/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-kube-api-access-lgr5z\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:50:09.406963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.406797 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d128ee21-8e4e-498a-8ce6-daf49693054d-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:50:09.406963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.406805 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:50:09.406963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.406812 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d128ee21-8e4e-498a-8ce6-daf49693054d-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:50:09.406963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:09.406819 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:50:10.164626 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:10.164537 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6" Jul 16 17:50:10.165042 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:10.164537 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8" Jul 16 17:50:10.191863 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:10.191812 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8"] Jul 16 17:50:10.193573 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:10.193555 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6756b54b-fwrx8"] Jul 16 17:50:10.210792 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:10.210765 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6"] Jul 16 17:50:10.230521 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:10.230498 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-569dccd749-lgtr6"] Jul 16 17:50:11.876192 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:11.876161 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" path="/var/lib/kubelet/pods/0e67c3b8-e9aa-403e-b6d9-dff5c5db848b/volumes" Jul 16 17:50:11.876673 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:11.876658 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" path="/var/lib/kubelet/pods/d128ee21-8e4e-498a-8ce6-daf49693054d/volumes" Jul 16 17:50:24.773849 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.773815 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj"] Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774163 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="feast-init" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774176 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="feast-init" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774186 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659b7892-52af-4545-8f67-ca6cc4296750" containerName="postgres" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774192 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="659b7892-52af-4545-8f67-ca6cc4296750" containerName="postgres" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774198 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="feast-init" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774204 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="feast-init" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774212 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="registry" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774218 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="registry" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774230 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f3e31b-1d28-4adf-b928-bcd5b5f14412" containerName="redis" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774235 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f3e31b-1d28-4adf-b928-bcd5b5f14412" containerName="redis" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774241 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="feast-apply" Jul 16 17:50:24.774240 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774246 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="feast-apply" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774251 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="registry" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774256 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="registry" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774264 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="feast-apply" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774269 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="feast-apply" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774276 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="online" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774280 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="online" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774290 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="online" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774295 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="online" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774300 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341402da-904c-44a4-b227-dcc85952c9cc" containerName="manager" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774305 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="341402da-904c-44a4-b227-dcc85952c9cc" containerName="manager" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774353 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="registry" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774360 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="341402da-904c-44a4-b227-dcc85952c9cc" containerName="manager" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774366 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="registry" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774372 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e67c3b8-e9aa-403e-b6d9-dff5c5db848b" containerName="online" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774377 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="659b7892-52af-4545-8f67-ca6cc4296750" containerName="postgres" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774383 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f3e31b-1d28-4adf-b928-bcd5b5f14412" containerName="redis" Jul 16 17:50:24.774593 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.774389 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d128ee21-8e4e-498a-8ce6-daf49693054d" containerName="online" Jul 16 17:50:24.778905 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.778889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.783431 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.783409 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 16 17:50:24.783658 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.783642 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 16 17:50:24.783726 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.783669 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 16 17:50:24.783726 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.783683 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 16 17:50:24.783807 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.783676 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-5tgwg\"" Jul 16 17:50:24.784587 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.784570 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 16 17:50:24.784807 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.784791 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 16 17:50:24.808681 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.808652 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj"] Jul 16 17:50:24.827584 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.827554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.827584 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.827582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d8dd8951-fce5-4a50-a198-650509d596d8-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.827745 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.827604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.827745 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.827627 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.827815 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.827740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.827815 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.827770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbmd\" (UniqueName: \"kubernetes.io/projected/d8dd8951-fce5-4a50-a198-650509d596d8-kube-api-access-qqbmd\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.929050 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.929017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.929210 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.929083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.929278 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.929245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbmd\" (UniqueName: \"kubernetes.io/projected/d8dd8951-fce5-4a50-a198-650509d596d8-kube-api-access-qqbmd\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.929330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.929299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.929330 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.929318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d8dd8951-fce5-4a50-a198-650509d596d8-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.929477 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.929348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.929477 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:50:24.929464 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 16 17:50:24.929592 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:50:24.929465 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 16 17:50:24.929592 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:50:24.929545 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-offline-tls podName:d8dd8951-fce5-4a50-a198-650509d596d8 nodeName:}" failed. No retries permitted until 2026-07-16 17:50:25.429523739 +0000 UTC m=+1206.100993921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-offline-tls") pod "feast-simple-feast-setup-796bc5bb56-s9bbj" (UID: "d8dd8951-fce5-4a50-a198-650509d596d8") : secret "feast-simple-feast-setup-offline-tls" not found Jul 16 17:50:24.929592 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:50:24.929562 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-online-tls podName:d8dd8951-fce5-4a50-a198-650509d596d8 nodeName:}" failed. No retries permitted until 2026-07-16 17:50:25.429554513 +0000 UTC m=+1206.101024699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-online-tls") pod "feast-simple-feast-setup-796bc5bb56-s9bbj" (UID: "d8dd8951-fce5-4a50-a198-650509d596d8") : secret "feast-simple-feast-setup-online-tls" not found Jul 16 17:50:24.930048 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.930024 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d8dd8951-fce5-4a50-a198-650509d596d8-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.931642 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.931623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.931722 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.931640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:24.943671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:24.943649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbmd\" (UniqueName: \"kubernetes.io/projected/d8dd8951-fce5-4a50-a198-650509d596d8-kube-api-access-qqbmd\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:25.434186 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:25.434150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:25.434374 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:25.434194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:25.436690 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:25.436659 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:25.436809 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:25.436711 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-s9bbj\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:25.702771 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:25.702670 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:25.852315 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:25.852285 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj"] Jul 16 17:50:25.855308 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:50:25.855278 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8dd8951_fce5_4a50_a198_650509d596d8.slice/crio-0174d749a4d73988304e11de6bdc8b58ad619139a1879a13e47b14084d9df341 WatchSource:0}: Error finding container 0174d749a4d73988304e11de6bdc8b58ad619139a1879a13e47b14084d9df341: Status 404 returned error can't find the container with id 0174d749a4d73988304e11de6bdc8b58ad619139a1879a13e47b14084d9df341 Jul 16 17:50:25.857145 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:25.857130 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:50:26.231586 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:26.231549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerStarted","Data":"0174d749a4d73988304e11de6bdc8b58ad619139a1879a13e47b14084d9df341"} Jul 16 17:50:42.314320 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:42.314281 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerStarted","Data":"3221e7c2593e166299fd4b81e53dc162519b64e82c3f0692925ea4588727eac7"} Jul 16 17:50:46.333684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:46.333599 2565 generic.go:356] "Generic (PLEG): container finished" podID="d8dd8951-fce5-4a50-a198-650509d596d8" containerID="3221e7c2593e166299fd4b81e53dc162519b64e82c3f0692925ea4588727eac7" exitCode=0 Jul 16 17:50:46.333684 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:46.333649 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerDied","Data":"3221e7c2593e166299fd4b81e53dc162519b64e82c3f0692925ea4588727eac7"} Jul 16 17:50:47.340346 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:47.340309 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerStarted","Data":"c2092d0b8f7240e697f7310ab2657b24dd35086d34aa8ebb7d049cde24635f36"} Jul 16 17:50:52.363537 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:52.363502 2565 generic.go:356] "Generic (PLEG): container finished" podID="d8dd8951-fce5-4a50-a198-650509d596d8" containerID="c2092d0b8f7240e697f7310ab2657b24dd35086d34aa8ebb7d049cde24635f36" exitCode=0 Jul 16 17:50:52.363935 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:52.363561 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerDied","Data":"c2092d0b8f7240e697f7310ab2657b24dd35086d34aa8ebb7d049cde24635f36"} Jul 16 17:50:53.374893 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:53.374819 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerStarted","Data":"2f253562f532c830b144785720b211d567ba3452ea457e4be319bee659a73859"} Jul 16 17:50:53.375397 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:53.374924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerStarted","Data":"e23f021c7a0ae202cc9a93e18625bf67e61b7fdb2fb70c071939a33a2cfaa3b2"} Jul 16 17:50:53.375397 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:53.374946 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerStarted","Data":"67e7d3db8da5f28b99f7b4d4f7a6428c0527fd89fcbda42c9a9275729d155a9e"} Jul 16 17:50:53.375397 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:53.374963 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerStarted","Data":"9e236e526b97239289cfa07039e1b39ce8860c83b80e35e6bddc05883dc82def"} Jul 16 17:50:53.412360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:53.412283 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podStartSLOduration=13.969268038 podStartE2EDuration="29.412257851s" totalImagesPullingTime="15.442989813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:50:53.409085051 +0000 UTC m=+1234.080555257" watchObservedRunningTime="2026-07-16 17:50:53.412257851 +0000 UTC m=+1234.083728055" Jul 16 17:50:55.702800 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.702760 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:55.703434 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.703408 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:55.703570 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.703442 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:55.703570 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.703472 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:55.703570 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.703488 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:55.703570 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.703508 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:55.703570 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.703525 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:55.703570 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.703544 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:55.705374 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.705331 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.38:8816: connect: connection refused" Jul 16 17:50:55.705374 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.705361 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused" Jul 16 17:50:55.705561 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.705373 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Jul 16 17:50:55.705561 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:55.705435 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Jul 16 17:50:58.703932 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:58.703866 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Jul 16 17:50:58.704964 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:58.704925 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:58.705134 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:58.705121 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:58.705780 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:58.705759 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:58.706058 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:58.706039 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:58.709855 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:58.709827 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:50:59.413884 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:50:59.413858 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:51:01.704102 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:51:01.704065 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:51:02.423000 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:51:02.422971 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:52:05.240881 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.240831 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj"] Jul 16 17:52:05.241469 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.241088 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="registry" containerID="cri-o://9e236e526b97239289cfa07039e1b39ce8860c83b80e35e6bddc05883dc82def" gracePeriod=30 Jul 16 17:52:05.241469 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.241214 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="offline" containerID="cri-o://e23f021c7a0ae202cc9a93e18625bf67e61b7fdb2fb70c071939a33a2cfaa3b2" gracePeriod=30 Jul 16 17:52:05.241469 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.241228 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="online" containerID="cri-o://67e7d3db8da5f28b99f7b4d4f7a6428c0527fd89fcbda42c9a9275729d155a9e" gracePeriod=30 Jul 16 17:52:05.241469 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.241355 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="ui" containerID="cri-o://2f253562f532c830b144785720b211d567ba3452ea457e4be319bee659a73859" gracePeriod=30 Jul 16 17:52:05.605831 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.605791 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk"] Jul 16 17:52:05.610794 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.610771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.613823 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.613797 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-pqmwf\"" Jul 16 17:52:05.629032 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.629005 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk"] Jul 16 17:52:05.704898 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.704868 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5c984c94-d369-4c06-ac65-670b5938ce29-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.704898 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.704900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.705144 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.704943 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8rk\" (UniqueName: \"kubernetes.io/projected/5c984c94-d369-4c06-ac65-670b5938ce29-kube-api-access-nl8rk\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.705144 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.705060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.705144 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.705099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.705263 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.705167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.806208 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.806171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8rk\" (UniqueName: \"kubernetes.io/projected/5c984c94-d369-4c06-ac65-670b5938ce29-kube-api-access-nl8rk\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.806419 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.806247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.806419 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.806277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.806419 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.806311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.806419 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.806349 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5c984c94-d369-4c06-ac65-670b5938ce29-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.806419 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.806372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.806419 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:52:05.806393 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 16 17:52:05.806741 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:52:05.806433 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 16 17:52:05.806741 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:52:05.806464 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-online-tls podName:5c984c94-d369-4c06-ac65-670b5938ce29 nodeName:}" failed. No retries permitted until 2026-07-16 17:52:06.306445212 +0000 UTC m=+1306.977915410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-online-tls") pod "feast-simple-feast-setup-796bc5bb56-dv9jk" (UID: "5c984c94-d369-4c06-ac65-670b5938ce29") : secret "feast-simple-feast-setup-online-tls" not found Jul 16 17:52:05.806741 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:52:05.806506 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-ui-tls podName:5c984c94-d369-4c06-ac65-670b5938ce29 nodeName:}" failed. No retries permitted until 2026-07-16 17:52:06.306486314 +0000 UTC m=+1306.977956525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-ui-tls") pod "feast-simple-feast-setup-796bc5bb56-dv9jk" (UID: "5c984c94-d369-4c06-ac65-670b5938ce29") : secret "feast-simple-feast-setup-ui-tls" not found Jul 16 17:52:05.806891 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.806794 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5c984c94-d369-4c06-ac65-670b5938ce29-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.808765 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.808743 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.808946 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.808926 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:05.817729 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:05.817706 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8rk\" (UniqueName: \"kubernetes.io/projected/5c984c94-d369-4c06-ac65-670b5938ce29-kube-api-access-nl8rk\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:06.310368 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.310318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:06.310873 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.310426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:06.312881 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.312857 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:06.312942 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.312896 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-dv9jk\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:06.520925 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.520880 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:06.661035 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:52:06.661005 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c984c94_d369_4c06_ac65_670b5938ce29.slice/crio-f321547e0e9e53f628dfc2abd791633a5c58f051f47469a3242be7cd66dcba5c WatchSource:0}: Error finding container f321547e0e9e53f628dfc2abd791633a5c58f051f47469a3242be7cd66dcba5c: Status 404 returned error can't find the container with id f321547e0e9e53f628dfc2abd791633a5c58f051f47469a3242be7cd66dcba5c Jul 16 17:52:06.661137 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.661064 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk"] Jul 16 17:52:06.700067 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.699996 2565 generic.go:356] "Generic (PLEG): container finished" podID="d8dd8951-fce5-4a50-a198-650509d596d8" containerID="2f253562f532c830b144785720b211d567ba3452ea457e4be319bee659a73859" exitCode=0 Jul 16 17:52:06.700067 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.700067 2565 generic.go:356] "Generic (PLEG): container finished" podID="d8dd8951-fce5-4a50-a198-650509d596d8" containerID="67e7d3db8da5f28b99f7b4d4f7a6428c0527fd89fcbda42c9a9275729d155a9e" exitCode=0 Jul 16 17:52:06.700266 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.700070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerDied","Data":"2f253562f532c830b144785720b211d567ba3452ea457e4be319bee659a73859"} Jul 16 17:52:06.700266 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.700134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerDied","Data":"67e7d3db8da5f28b99f7b4d4f7a6428c0527fd89fcbda42c9a9275729d155a9e"} Jul 16 17:52:06.701373 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:06.701351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerStarted","Data":"f321547e0e9e53f628dfc2abd791633a5c58f051f47469a3242be7cd66dcba5c"} Jul 16 17:52:07.707245 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:07.707208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerStarted","Data":"bd9fdbfe48be8cd4f6bc42af572a7d6ccfc913771613d51076778b61120302d6"} Jul 16 17:52:09.411549 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:09.411506 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Jul 16 17:52:11.728139 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:11.728104 2565 generic.go:356] "Generic (PLEG): container finished" podID="5c984c94-d369-4c06-ac65-670b5938ce29" containerID="bd9fdbfe48be8cd4f6bc42af572a7d6ccfc913771613d51076778b61120302d6" exitCode=0 Jul 16 17:52:11.728505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:11.728176 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerDied","Data":"bd9fdbfe48be8cd4f6bc42af572a7d6ccfc913771613d51076778b61120302d6"} Jul 16 17:52:12.423128 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:12.423085 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Jul 16 17:52:12.734104 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:12.734021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerStarted","Data":"0bbf865151b66eee3d82fbe436c9c8a8aac96b4710c1307163ee7d2d40bbd103"} Jul 16 17:52:17.759908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:17.759807 2565 generic.go:356] "Generic (PLEG): container finished" podID="5c984c94-d369-4c06-ac65-670b5938ce29" containerID="0bbf865151b66eee3d82fbe436c9c8a8aac96b4710c1307163ee7d2d40bbd103" exitCode=0 Jul 16 17:52:17.759908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:17.759876 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerDied","Data":"0bbf865151b66eee3d82fbe436c9c8a8aac96b4710c1307163ee7d2d40bbd103"} Jul 16 17:52:18.772092 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:18.772051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerStarted","Data":"98d4cda10951e2c885b652a12b361fe7ec86079d57f7bf2ea502107a5c7db729"} Jul 16 17:52:18.772581 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:18.772129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerStarted","Data":"bb52b6ea9e09e9baa0fb420b00af4e08a460cd7609c1eddcf697dca796f038ee"} Jul 16 17:52:18.772581 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:18.772150 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerStarted","Data":"c1f369c607161340ae4592a8ad710210f91b73484957eca09f7f25b2cc0ab794"} Jul 16 17:52:18.772581 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:18.772179 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerStarted","Data":"2b7ad087482b2723e42cb5607f7da259ee8a03f54c3decd5fd487234e34df000"} Jul 16 17:52:18.805608 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:18.805544 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podStartSLOduration=13.805522618 podStartE2EDuration="13.805522618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:18.803217136 +0000 UTC m=+1319.474687343" watchObservedRunningTime="2026-07-16 17:52:18.805522618 +0000 UTC m=+1319.476992823" Jul 16 17:52:19.411947 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:19.411893 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Jul 16 17:52:21.521551 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:21.521470 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:21.522095 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:21.521634 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:21.522095 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:21.521670 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:21.522095 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:21.521688 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:21.523750 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:21.523708 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.39:8816: connect: connection refused" Jul 16 17:52:21.523750 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:21.523737 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Jul 16 17:52:21.523961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:21.523785 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Jul 16 17:52:21.523961 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:21.523817 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.39:6571: connect: connection refused" Jul 16 17:52:22.422765 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:22.422714 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Jul 16 17:52:24.522118 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.522089 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.522603 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.522299 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.522603 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.522568 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.523026 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.522999 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.523147 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.523050 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.523147 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.523070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.523367 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.523348 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.523424 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.523416 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.523465 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.523456 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.527505 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.527483 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.808223 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.808196 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:24.811535 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:24.811513 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:52:29.411586 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:29.411545 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Jul 16 17:52:29.411982 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:29.411725 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:52:32.422674 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:32.422632 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Jul 16 17:52:32.423091 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:32.422744 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:52:35.862755 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:35.862723 2565 generic.go:356] "Generic (PLEG): container finished" podID="d8dd8951-fce5-4a50-a198-650509d596d8" containerID="e23f021c7a0ae202cc9a93e18625bf67e61b7fdb2fb70c071939a33a2cfaa3b2" exitCode=137 Jul 16 17:52:35.862755 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:35.862751 2565 generic.go:356] "Generic (PLEG): container finished" podID="d8dd8951-fce5-4a50-a198-650509d596d8" containerID="9e236e526b97239289cfa07039e1b39ce8860c83b80e35e6bddc05883dc82def" exitCode=137 Jul 16 17:52:35.863152 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:35.862795 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerDied","Data":"e23f021c7a0ae202cc9a93e18625bf67e61b7fdb2fb70c071939a33a2cfaa3b2"} Jul 16 17:52:35.863152 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:35.862856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerDied","Data":"9e236e526b97239289cfa07039e1b39ce8860c83b80e35e6bddc05883dc82def"} Jul 16 17:52:35.903926 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:35.903900 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:52:36.076937 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.076905 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqbmd\" (UniqueName: \"kubernetes.io/projected/d8dd8951-fce5-4a50-a198-650509d596d8-kube-api-access-qqbmd\") pod \"d8dd8951-fce5-4a50-a198-650509d596d8\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " Jul 16 17:52:36.077119 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.076973 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-offline-tls\") pod \"d8dd8951-fce5-4a50-a198-650509d596d8\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " Jul 16 17:52:36.077119 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.077010 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d8dd8951-fce5-4a50-a198-650509d596d8-feast-data\") pod \"d8dd8951-fce5-4a50-a198-650509d596d8\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " Jul 16 17:52:36.077119 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.077037 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-ui-tls\") pod \"d8dd8951-fce5-4a50-a198-650509d596d8\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " Jul 16 17:52:36.077119 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.077063 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-online-tls\") pod \"d8dd8951-fce5-4a50-a198-650509d596d8\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " Jul 16 17:52:36.077119 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.077080 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-registry-tls\") pod \"d8dd8951-fce5-4a50-a198-650509d596d8\" (UID: \"d8dd8951-fce5-4a50-a198-650509d596d8\") " Jul 16 17:52:36.077650 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.077604 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8dd8951-fce5-4a50-a198-650509d596d8-feast-data" (OuterVolumeSpecName: "feast-data") pod "d8dd8951-fce5-4a50-a198-650509d596d8" (UID: "d8dd8951-fce5-4a50-a198-650509d596d8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:36.079547 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.079516 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-online-tls" (OuterVolumeSpecName: "online-tls") pod "d8dd8951-fce5-4a50-a198-650509d596d8" (UID: "d8dd8951-fce5-4a50-a198-650509d596d8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:52:36.079642 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.079538 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d8dd8951-fce5-4a50-a198-650509d596d8" (UID: "d8dd8951-fce5-4a50-a198-650509d596d8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:52:36.079642 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.079628 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d8dd8951-fce5-4a50-a198-650509d596d8" (UID: "d8dd8951-fce5-4a50-a198-650509d596d8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:52:36.079728 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.079692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d8dd8951-fce5-4a50-a198-650509d596d8" (UID: "d8dd8951-fce5-4a50-a198-650509d596d8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:52:36.079728 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.079701 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dd8951-fce5-4a50-a198-650509d596d8-kube-api-access-qqbmd" (OuterVolumeSpecName: "kube-api-access-qqbmd") pod "d8dd8951-fce5-4a50-a198-650509d596d8" (UID: "d8dd8951-fce5-4a50-a198-650509d596d8"). InnerVolumeSpecName "kube-api-access-qqbmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:52:36.177903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.177868 2565 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-ui-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:52:36.177903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.177897 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:52:36.177903 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.177908 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:52:36.178210 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.177917 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbmd\" (UniqueName: \"kubernetes.io/projected/d8dd8951-fce5-4a50-a198-650509d596d8-kube-api-access-qqbmd\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:52:36.178210 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.177925 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d8dd8951-fce5-4a50-a198-650509d596d8-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:52:36.178210 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.177933 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d8dd8951-fce5-4a50-a198-650509d596d8-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:52:36.869706 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.869672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" event={"ID":"d8dd8951-fce5-4a50-a198-650509d596d8","Type":"ContainerDied","Data":"0174d749a4d73988304e11de6bdc8b58ad619139a1879a13e47b14084d9df341"} Jul 16 17:52:36.870117 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.869730 2565 scope.go:117] "RemoveContainer" containerID="2f253562f532c830b144785720b211d567ba3452ea457e4be319bee659a73859" Jul 16 17:52:36.870117 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.869740 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj" Jul 16 17:52:36.879547 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.879529 2565 scope.go:117] "RemoveContainer" containerID="e23f021c7a0ae202cc9a93e18625bf67e61b7fdb2fb70c071939a33a2cfaa3b2" Jul 16 17:52:36.887200 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.887185 2565 scope.go:117] "RemoveContainer" containerID="67e7d3db8da5f28b99f7b4d4f7a6428c0527fd89fcbda42c9a9275729d155a9e" Jul 16 17:52:36.894618 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.894594 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj"] Jul 16 17:52:36.895084 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.895069 2565 scope.go:117] "RemoveContainer" containerID="9e236e526b97239289cfa07039e1b39ce8860c83b80e35e6bddc05883dc82def" Jul 16 17:52:36.901333 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.901311 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-s9bbj"] Jul 16 17:52:36.903250 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.903231 2565 scope.go:117] "RemoveContainer" containerID="c2092d0b8f7240e697f7310ab2657b24dd35086d34aa8ebb7d049cde24635f36" Jul 16 17:52:36.910826 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:36.910808 2565 scope.go:117] "RemoveContainer" containerID="3221e7c2593e166299fd4b81e53dc162519b64e82c3f0692925ea4588727eac7" Jul 16 17:52:37.876692 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:52:37.876651 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" path="/var/lib/kubelet/pods/d8dd8951-fce5-4a50-a198-650509d596d8/volumes" Jul 16 17:53:27.644810 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.644770 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk"] Jul 16 17:53:27.645360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645273 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="online" Jul 16 17:53:27.645360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645292 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="online" Jul 16 17:53:27.645360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645310 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="registry" Jul 16 17:53:27.645360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645319 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="registry" Jul 16 17:53:27.645360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645338 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="offline" Jul 16 17:53:27.645360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645347 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="offline" Jul 16 17:53:27.645360 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645362 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="feast-init" Jul 16 17:53:27.645676 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645373 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="feast-init" Jul 16 17:53:27.645676 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645382 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="feast-apply" Jul 16 17:53:27.645676 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645391 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="feast-apply" Jul 16 17:53:27.645676 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645404 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="ui" Jul 16 17:53:27.645676 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645413 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="ui" Jul 16 17:53:27.645676 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645506 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="registry" Jul 16 17:53:27.645676 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645515 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="online" Jul 16 17:53:27.645676 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645535 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="ui" Jul 16 17:53:27.645676 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.645547 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8dd8951-fce5-4a50-a198-650509d596d8" containerName="offline" Jul 16 17:53:27.650338 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.650322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.653316 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.653297 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 16 17:53:27.653582 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.653561 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 16 17:53:27.654074 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.654057 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 16 17:53:27.654157 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.654081 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 16 17:53:27.654376 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.654355 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 16 17:53:27.654590 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.654572 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 16 17:53:27.654651 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.654616 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-g8ktx\"" Jul 16 17:53:27.665637 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.665616 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk"] Jul 16 17:53:27.821086 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.821044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-offline-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.821291 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.821161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-online-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.821291 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.821199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-registry-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.821291 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.821224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-feast-data\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.821291 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.821252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72b4r\" (UniqueName: \"kubernetes.io/projected/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-kube-api-access-72b4r\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.821291 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.821279 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-ui-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.922035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.921942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72b4r\" (UniqueName: \"kubernetes.io/projected/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-kube-api-access-72b4r\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.922035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.921979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-ui-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.922035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.922005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-offline-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.922393 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.922066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-online-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.922393 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.922087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-registry-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.922393 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.922105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-feast-data\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.922580 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.922559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-feast-data\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.923153 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.923128 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-registry-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.924554 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.924525 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-online-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.924671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.924587 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-ui-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.924671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.924621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-offline-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.932388 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.932365 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72b4r\" (UniqueName: \"kubernetes.io/projected/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-kube-api-access-72b4r\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:27.959355 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:27.959316 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:28.102239 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:28.102212 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk"] Jul 16 17:53:28.104656 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:53:28.104624 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb399c91f_f4b8_4c5d_9601_eb8eec60d0bd.slice/crio-22adb632cac2fb58359add89e65e5826a64c214519d67b06bb2431e303580cea WatchSource:0}: Error finding container 22adb632cac2fb58359add89e65e5826a64c214519d67b06bb2431e303580cea: Status 404 returned error can't find the container with id 22adb632cac2fb58359add89e65e5826a64c214519d67b06bb2431e303580cea Jul 16 17:53:29.094156 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:29.094118 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerStarted","Data":"100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b"} Jul 16 17:53:29.094531 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:29.094172 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerStarted","Data":"22adb632cac2fb58359add89e65e5826a64c214519d67b06bb2431e303580cea"} Jul 16 17:53:33.112864 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:33.112814 2565 generic.go:356] "Generic (PLEG): container finished" podID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerID="100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b" exitCode=0 Jul 16 17:53:33.113232 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:33.112916 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerDied","Data":"100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b"} Jul 16 17:53:34.118971 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:34.118935 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerStarted","Data":"a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3"} Jul 16 17:53:39.143955 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:39.143923 2565 generic.go:356] "Generic (PLEG): container finished" podID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerID="a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3" exitCode=0 Jul 16 17:53:39.144378 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:39.143951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerDied","Data":"a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3"} Jul 16 17:53:40.153306 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:40.153257 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerStarted","Data":"d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752"} Jul 16 17:53:40.153818 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:40.153329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerStarted","Data":"88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33"} Jul 16 17:53:40.153818 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:40.153349 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerStarted","Data":"fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30"} Jul 16 17:53:40.183235 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:40.183182 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podStartSLOduration=13.183166231 podStartE2EDuration="13.183166231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:53:40.182107747 +0000 UTC m=+1400.853577952" watchObservedRunningTime="2026-07-16 17:53:40.183166231 +0000 UTC m=+1400.854636434" Jul 16 17:53:42.960074 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:42.960033 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:42.960589 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:42.960113 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:42.960589 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:42.960131 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:42.961908 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:42.961878 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.40:8816: connect: connection refused" Jul 16 17:53:42.962068 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:42.961921 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused" Jul 16 17:53:42.962068 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:42.961940 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused" Jul 16 17:53:45.960723 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:45.960695 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:45.961189 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:45.960811 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:45.961279 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:45.961212 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:45.961279 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:45.961248 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:45.961576 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:45.961554 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:45.961672 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:45.961655 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:45.966798 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:45.966779 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:46.187262 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:46.187231 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:53:46.191874 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:53:46.191834 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:54:20.764946 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:20.764872 2565 scope.go:117] "RemoveContainer" containerID="c6d3ab44e9bf4a0ff3c3b304f5d85d56de5ed51d9e7ceda8d3e5aac00b69284b" Jul 16 17:54:20.773442 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:20.773418 2565 scope.go:117] "RemoveContainer" containerID="121dc4b70e52f575f13c5bbf84753b84b442d17f13d63f2f7db72a8d52480314" Jul 16 17:54:20.781384 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:20.781365 2565 scope.go:117] "RemoveContainer" containerID="d3af612f94b6007d75d265d88700d103ad250c67f49efa38014650b0e5dba59e" Jul 16 17:54:20.789603 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:20.789579 2565 scope.go:117] "RemoveContainer" containerID="cff41bdc7effadc4580fcd266fc36707aab9eaf76b50ae27d5012b5d86a8a037" Jul 16 17:54:20.800030 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:20.800000 2565 scope.go:117] "RemoveContainer" containerID="fe3a4f663fc25912c408d238738823a4334125a6f4d3acc39d99643e850e06bc" Jul 16 17:54:20.807937 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:20.807916 2565 scope.go:117] "RemoveContainer" containerID="f02c1ade4676ee6ff34be9a611987b41f8c2a77490cc7f9f0ce7499f5b58ad84" Jul 16 17:54:20.815113 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:20.815085 2565 scope.go:117] "RemoveContainer" containerID="0e030c60ca24c03f61289bd9163f1066489446eecc075f401d37faee4a8996f5" Jul 16 17:54:20.821964 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:20.821946 2565 scope.go:117] "RemoveContainer" containerID="236bbd68cef1c5e6373e460efbc03bf6503d866d996a0316be78166fad7497c6" Jul 16 17:54:48.944347 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:48.944312 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk"] Jul 16 17:54:48.944786 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:48.944572 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="ui" containerID="cri-o://d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752" gracePeriod=30 Jul 16 17:54:48.944786 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:48.944625 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="offline" containerID="cri-o://88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33" gracePeriod=30 Jul 16 17:54:48.944786 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:48.944548 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="online" containerID="cri-o://fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30" gracePeriod=30 Jul 16 17:54:49.156674 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:49.156638 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk"] Jul 16 17:54:49.156989 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:49.156959 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="registry" containerID="cri-o://2b7ad087482b2723e42cb5607f7da259ee8a03f54c3decd5fd487234e34df000" gracePeriod=30 Jul 16 17:54:49.157132 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:49.157070 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="ui" containerID="cri-o://98d4cda10951e2c885b652a12b361fe7ec86079d57f7bf2ea502107a5c7db729" gracePeriod=30 Jul 16 17:54:49.157132 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:49.157079 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="online" containerID="cri-o://c1f369c607161340ae4592a8ad710210f91b73484957eca09f7f25b2cc0ab794" gracePeriod=30 Jul 16 17:54:49.157341 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:49.157074 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="offline" containerID="cri-o://bb52b6ea9e09e9baa0fb420b00af4e08a460cd7609c1eddcf697dca796f038ee" gracePeriod=30 Jul 16 17:54:50.476699 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:50.476662 2565 generic.go:356] "Generic (PLEG): container finished" podID="5c984c94-d369-4c06-ac65-670b5938ce29" containerID="98d4cda10951e2c885b652a12b361fe7ec86079d57f7bf2ea502107a5c7db729" exitCode=0 Jul 16 17:54:50.476699 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:50.476684 2565 generic.go:356] "Generic (PLEG): container finished" podID="5c984c94-d369-4c06-ac65-670b5938ce29" containerID="c1f369c607161340ae4592a8ad710210f91b73484957eca09f7f25b2cc0ab794" exitCode=0 Jul 16 17:54:50.477169 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:50.476732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerDied","Data":"98d4cda10951e2c885b652a12b361fe7ec86079d57f7bf2ea502107a5c7db729"} Jul 16 17:54:50.477169 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:50.476781 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerDied","Data":"c1f369c607161340ae4592a8ad710210f91b73484957eca09f7f25b2cc0ab794"} Jul 16 17:54:50.479256 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:50.479237 2565 generic.go:356] "Generic (PLEG): container finished" podID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerID="d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752" exitCode=0 Jul 16 17:54:50.479256 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:50.479252 2565 generic.go:356] "Generic (PLEG): container finished" podID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerID="fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30" exitCode=0 Jul 16 17:54:50.479403 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:50.479309 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerDied","Data":"d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752"} Jul 16 17:54:50.479403 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:50.479349 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerDied","Data":"fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30"} Jul 16 17:54:54.523799 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:54.523759 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Jul 16 17:54:54.809058 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:54.809020 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Jul 16 17:54:55.962148 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:55.962095 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused" Jul 16 17:54:56.187566 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:54:56.187526 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused" Jul 16 17:55:04.523272 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:04.523225 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Jul 16 17:55:04.809378 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:04.809335 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Jul 16 17:55:05.961634 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:05.961593 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused" Jul 16 17:55:06.187829 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:06.187785 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused" Jul 16 17:55:14.523963 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:14.523910 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Jul 16 17:55:14.524429 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:14.524097 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:55:14.809239 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:14.809197 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Jul 16 17:55:14.809433 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:14.809355 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:55:15.962117 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:15.962070 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused" Jul 16 17:55:15.962520 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:15.962208 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:55:16.188132 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:16.188087 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused" Jul 16 17:55:16.188306 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:16.188218 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:55:19.139154 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.139129 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:55:19.274822 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.274743 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-registry-tls\") pod \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " Jul 16 17:55:19.274822 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.274794 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72b4r\" (UniqueName: \"kubernetes.io/projected/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-kube-api-access-72b4r\") pod \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " Jul 16 17:55:19.274822 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.274818 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-feast-data\") pod \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " Jul 16 17:55:19.275082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.274889 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-ui-tls\") pod \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " Jul 16 17:55:19.275082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.274918 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-offline-tls\") pod \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " Jul 16 17:55:19.275082 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.274956 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-online-tls\") pod \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\" (UID: \"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd\") " Jul 16 17:55:19.275218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.275170 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" (UID: "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:55:19.275280 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.275241 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.275396 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.275372 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-feast-data" (OuterVolumeSpecName: "feast-data") pod "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" (UID: "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:55:19.277295 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.277264 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-kube-api-access-72b4r" (OuterVolumeSpecName: "kube-api-access-72b4r") pod "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" (UID: "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd"). InnerVolumeSpecName "kube-api-access-72b4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:55:19.277295 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.277282 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-online-tls" (OuterVolumeSpecName: "online-tls") pod "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" (UID: "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:55:19.277451 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.277308 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" (UID: "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:55:19.277451 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.277341 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" (UID: "b399c91f-f4b8-4c5d-9601-eb8eec60d0bd"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:55:19.376304 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.376268 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.376304 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.376295 2565 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-ui-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.376304 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.376303 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.376304 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.376310 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.376552 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.376319 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72b4r\" (UniqueName: \"kubernetes.io/projected/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd-kube-api-access-72b4r\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.601264 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.601231 2565 generic.go:356] "Generic (PLEG): container finished" podID="5c984c94-d369-4c06-ac65-670b5938ce29" containerID="bb52b6ea9e09e9baa0fb420b00af4e08a460cd7609c1eddcf697dca796f038ee" exitCode=137 Jul 16 17:55:19.601264 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.601255 2565 generic.go:356] "Generic (PLEG): container finished" podID="5c984c94-d369-4c06-ac65-670b5938ce29" containerID="2b7ad087482b2723e42cb5607f7da259ee8a03f54c3decd5fd487234e34df000" exitCode=137 Jul 16 17:55:19.601498 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.601306 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerDied","Data":"bb52b6ea9e09e9baa0fb420b00af4e08a460cd7609c1eddcf697dca796f038ee"} Jul 16 17:55:19.601498 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.601355 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerDied","Data":"2b7ad087482b2723e42cb5607f7da259ee8a03f54c3decd5fd487234e34df000"} Jul 16 17:55:19.603522 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.603502 2565 generic.go:356] "Generic (PLEG): container finished" podID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerID="88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33" exitCode=137 Jul 16 17:55:19.603652 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.603534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerDied","Data":"88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33"} Jul 16 17:55:19.603652 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.603557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" event={"ID":"b399c91f-f4b8-4c5d-9601-eb8eec60d0bd","Type":"ContainerDied","Data":"22adb632cac2fb58359add89e65e5826a64c214519d67b06bb2431e303580cea"} Jul 16 17:55:19.603652 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.603573 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk" Jul 16 17:55:19.603652 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.603582 2565 scope.go:117] "RemoveContainer" containerID="d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752" Jul 16 17:55:19.613116 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.613099 2565 scope.go:117] "RemoveContainer" containerID="88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33" Jul 16 17:55:19.620811 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.620794 2565 scope.go:117] "RemoveContainer" containerID="fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30" Jul 16 17:55:19.628107 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.628090 2565 scope.go:117] "RemoveContainer" containerID="a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3" Jul 16 17:55:19.630613 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.630590 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk"] Jul 16 17:55:19.635713 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.635689 2565 scope.go:117] "RemoveContainer" containerID="100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b" Jul 16 17:55:19.638703 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.638679 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-zhxnk"] Jul 16 17:55:19.642993 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.642965 2565 scope.go:117] "RemoveContainer" containerID="d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752" Jul 16 17:55:19.643237 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:55:19.643218 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752\": container with ID starting with d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752 not found: ID does not exist" containerID="d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752" Jul 16 17:55:19.643297 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.643254 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752"} err="failed to get container status \"d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752\": rpc error: code = NotFound desc = could not find container \"d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752\": container with ID starting with d71159dd70150878c4a27cb60b9a79132ba30e3f240a01dcb75391143ee58752 not found: ID does not exist" Jul 16 17:55:19.643297 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.643274 2565 scope.go:117] "RemoveContainer" containerID="88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33" Jul 16 17:55:19.643501 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:55:19.643483 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33\": container with ID starting with 88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33 not found: ID does not exist" containerID="88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33" Jul 16 17:55:19.643543 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.643507 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33"} err="failed to get container status \"88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33\": rpc error: code = NotFound desc = could not find container \"88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33\": container with ID starting with 88ee7aa894d4fade1c58a67d8e4a790e92e292ef5a1db6998ff990ee3b774b33 not found: ID does not exist" Jul 16 17:55:19.643543 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.643522 2565 scope.go:117] "RemoveContainer" containerID="fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30" Jul 16 17:55:19.643735 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:55:19.643719 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30\": container with ID starting with fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30 not found: ID does not exist" containerID="fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30" Jul 16 17:55:19.643775 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.643741 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30"} err="failed to get container status \"fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30\": rpc error: code = NotFound desc = could not find container \"fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30\": container with ID starting with fabded2d0b946a462d1e352df5ba78b95c5a1b92da1577a8c731ce2e49996d30 not found: ID does not exist" Jul 16 17:55:19.643775 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.643755 2565 scope.go:117] "RemoveContainer" containerID="a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3" Jul 16 17:55:19.644018 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:55:19.644002 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3\": container with ID starting with a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3 not found: ID does not exist" containerID="a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3" Jul 16 17:55:19.644063 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.644024 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3"} err="failed to get container status \"a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3\": rpc error: code = NotFound desc = could not find container \"a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3\": container with ID starting with a4ef5ac0cfc7008ff48edc60bfec8257fb0005cfe2cab96536308f77caaa6ba3 not found: ID does not exist" Jul 16 17:55:19.644063 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.644037 2565 scope.go:117] "RemoveContainer" containerID="100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b" Jul 16 17:55:19.644251 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:55:19.644233 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b\": container with ID starting with 100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b not found: ID does not exist" containerID="100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b" Jul 16 17:55:19.644293 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.644256 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b"} err="failed to get container status \"100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b\": rpc error: code = NotFound desc = could not find container \"100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b\": container with ID starting with 100db6feed44084d6bbb86bef46349c48f2b303c76e20bcf01384f33b844ff7b not found: ID does not exist" Jul 16 17:55:19.803171 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.803142 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:55:19.877314 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.877228 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" path="/var/lib/kubelet/pods/b399c91f-f4b8-4c5d-9601-eb8eec60d0bd/volumes" Jul 16 17:55:19.879599 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.879572 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-offline-tls\") pod \"5c984c94-d369-4c06-ac65-670b5938ce29\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " Jul 16 17:55:19.879733 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.879612 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5c984c94-d369-4c06-ac65-670b5938ce29-feast-data\") pod \"5c984c94-d369-4c06-ac65-670b5938ce29\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " Jul 16 17:55:19.879733 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.879639 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-registry-tls\") pod \"5c984c94-d369-4c06-ac65-670b5938ce29\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " Jul 16 17:55:19.879733 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.879677 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl8rk\" (UniqueName: \"kubernetes.io/projected/5c984c94-d369-4c06-ac65-670b5938ce29-kube-api-access-nl8rk\") pod \"5c984c94-d369-4c06-ac65-670b5938ce29\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " Jul 16 17:55:19.879733 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.879721 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-online-tls\") pod \"5c984c94-d369-4c06-ac65-670b5938ce29\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " Jul 16 17:55:19.879975 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.879756 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-ui-tls\") pod \"5c984c94-d369-4c06-ac65-670b5938ce29\" (UID: \"5c984c94-d369-4c06-ac65-670b5938ce29\") " Jul 16 17:55:19.880249 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.880222 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c984c94-d369-4c06-ac65-670b5938ce29-feast-data" (OuterVolumeSpecName: "feast-data") pod "5c984c94-d369-4c06-ac65-670b5938ce29" (UID: "5c984c94-d369-4c06-ac65-670b5938ce29"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:55:19.882218 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.882181 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5c984c94-d369-4c06-ac65-670b5938ce29" (UID: "5c984c94-d369-4c06-ac65-670b5938ce29"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:55:19.882493 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.882453 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5c984c94-d369-4c06-ac65-670b5938ce29" (UID: "5c984c94-d369-4c06-ac65-670b5938ce29"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:55:19.882591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.882505 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-online-tls" (OuterVolumeSpecName: "online-tls") pod "5c984c94-d369-4c06-ac65-670b5938ce29" (UID: "5c984c94-d369-4c06-ac65-670b5938ce29"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:55:19.882591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.882528 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c984c94-d369-4c06-ac65-670b5938ce29-kube-api-access-nl8rk" (OuterVolumeSpecName: "kube-api-access-nl8rk") pod "5c984c94-d369-4c06-ac65-670b5938ce29" (UID: "5c984c94-d369-4c06-ac65-670b5938ce29"). InnerVolumeSpecName "kube-api-access-nl8rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:55:19.882719 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.882675 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5c984c94-d369-4c06-ac65-670b5938ce29" (UID: "5c984c94-d369-4c06-ac65-670b5938ce29"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:55:19.980585 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.980560 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.980585 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.980584 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5c984c94-d369-4c06-ac65-670b5938ce29-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.980715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.980593 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.980715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.980602 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl8rk\" (UniqueName: \"kubernetes.io/projected/5c984c94-d369-4c06-ac65-670b5938ce29-kube-api-access-nl8rk\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.980715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.980634 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:19.980715 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:19.980644 2565 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5c984c94-d369-4c06-ac65-670b5938ce29-ui-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:55:20.610428 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.610394 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" event={"ID":"5c984c94-d369-4c06-ac65-670b5938ce29","Type":"ContainerDied","Data":"f321547e0e9e53f628dfc2abd791633a5c58f051f47469a3242be7cd66dcba5c"} Jul 16 17:55:20.610428 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.610413 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk" Jul 16 17:55:20.610972 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.610450 2565 scope.go:117] "RemoveContainer" containerID="98d4cda10951e2c885b652a12b361fe7ec86079d57f7bf2ea502107a5c7db729" Jul 16 17:55:20.620179 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.620157 2565 scope.go:117] "RemoveContainer" containerID="bb52b6ea9e09e9baa0fb420b00af4e08a460cd7609c1eddcf697dca796f038ee" Jul 16 17:55:20.627905 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.627887 2565 scope.go:117] "RemoveContainer" containerID="c1f369c607161340ae4592a8ad710210f91b73484957eca09f7f25b2cc0ab794" Jul 16 17:55:20.635512 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.635494 2565 scope.go:117] "RemoveContainer" containerID="2b7ad087482b2723e42cb5607f7da259ee8a03f54c3decd5fd487234e34df000" Jul 16 17:55:20.638806 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.638782 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk"] Jul 16 17:55:20.643395 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.643375 2565 scope.go:117] "RemoveContainer" containerID="0bbf865151b66eee3d82fbe436c9c8a8aac96b4710c1307163ee7d2d40bbd103" Jul 16 17:55:20.647021 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.646996 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-dv9jk"] Jul 16 17:55:20.650936 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:20.650919 2565 scope.go:117] "RemoveContainer" containerID="bd9fdbfe48be8cd4f6bc42af572a7d6ccfc913771613d51076778b61120302d6" Jul 16 17:55:21.876260 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:55:21.876229 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" path="/var/lib/kubelet/pods/5c984c94-d369-4c06-ac65-670b5938ce29/volumes" Jul 16 17:56:01.943372 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943278 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l"] Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943742 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="ui" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943760 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="ui" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943776 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="registry" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943784 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="registry" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943795 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="offline" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943804 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="offline" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943822 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="feast-init" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943830 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="feast-init" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943871 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="feast-init" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943880 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="feast-init" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943888 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="feast-apply" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943897 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="feast-apply" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943907 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="online" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943915 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="online" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943934 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="online" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943943 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="online" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943955 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="offline" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943963 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="offline" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943974 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="feast-apply" Jul 16 17:56:01.943979 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943982 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="feast-apply" Jul 16 17:56:01.945007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943992 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="ui" Jul 16 17:56:01.945007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.943999 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="ui" Jul 16 17:56:01.945007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.944078 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="online" Jul 16 17:56:01.945007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.944090 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="online" Jul 16 17:56:01.945007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.944101 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="offline" Jul 16 17:56:01.945007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.944109 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="ui" Jul 16 17:56:01.945007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.944117 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="registry" Jul 16 17:56:01.945007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.944127 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c984c94-d369-4c06-ac65-670b5938ce29" containerName="ui" Jul 16 17:56:01.945007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.944136 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b399c91f-f4b8-4c5d-9601-eb8eec60d0bd" containerName="offline" Jul 16 17:56:01.947999 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.947980 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:01.953140 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.953105 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 16 17:56:01.953485 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.953460 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 16 17:56:01.953600 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.953471 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 16 17:56:01.953600 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.953473 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 16 17:56:01.953740 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.953646 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-8w54q\"" Jul 16 17:56:01.953796 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.953766 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 16 17:56:01.953796 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.953773 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 16 17:56:01.965285 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:01.965264 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l"] Jul 16 17:56:02.008835 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.008804 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.009035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.008866 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.009035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.008941 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmzp\" (UniqueName: \"kubernetes.io/projected/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-kube-api-access-hlmzp\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.009035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.008977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.009035 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.009000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.009191 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.009050 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.110161 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.110124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.110161 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.110168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.110387 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.110299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.110387 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.110345 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmzp\" (UniqueName: \"kubernetes.io/projected/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-kube-api-access-hlmzp\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.110387 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.110377 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.110585 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.110558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.110662 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:56:02.110643 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 16 17:56:02.110731 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:56:02.110719 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-offline-tls podName:a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d nodeName:}" failed. No retries permitted until 2026-07-16 17:56:02.610701769 +0000 UTC m=+1543.282171951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-offline-tls") pod "feast-simple-feast-setup-796bc5bb56-n6b9l" (UID: "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d") : secret "feast-simple-feast-setup-offline-tls" not found Jul 16 17:56:02.110800 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.110575 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.112912 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.112891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.113020 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.112925 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.113020 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.113008 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.122264 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.122234 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmzp\" (UniqueName: \"kubernetes.io/projected/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-kube-api-access-hlmzp\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.614212 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.614160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.616894 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.616828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-n6b9l\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:02.877091 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:02.876985 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:03.034496 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:03.034470 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l"] Jul 16 17:56:03.036776 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:56:03.036749 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda27e21aa_a60c_41f1_a7ba_3e7ed214ad8d.slice/crio-20fb3f9b43c0ffedfdd0f573f39428cd049d99b54a05aac1481c12f97670837e WatchSource:0}: Error finding container 20fb3f9b43c0ffedfdd0f573f39428cd049d99b54a05aac1481c12f97670837e: Status 404 returned error can't find the container with id 20fb3f9b43c0ffedfdd0f573f39428cd049d99b54a05aac1481c12f97670837e Jul 16 17:56:03.791117 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:03.791075 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerStarted","Data":"63d0a3f17b8709d28a045123ce57ba4e2e71d9908822cda25a9efd0a5c362fc1"} Jul 16 17:56:03.791117 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:03.791118 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerStarted","Data":"20fb3f9b43c0ffedfdd0f573f39428cd049d99b54a05aac1481c12f97670837e"} Jul 16 17:56:07.809788 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:07.809755 2565 generic.go:356] "Generic (PLEG): container finished" podID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerID="63d0a3f17b8709d28a045123ce57ba4e2e71d9908822cda25a9efd0a5c362fc1" exitCode=0 Jul 16 17:56:07.810159 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:07.809805 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerDied","Data":"63d0a3f17b8709d28a045123ce57ba4e2e71d9908822cda25a9efd0a5c362fc1"} Jul 16 17:56:08.815691 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:08.815655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerStarted","Data":"94fdf741355c2a7abe8a7a980898ffb95dad18a93aa016b23c9a29cd8085330d"} Jul 16 17:56:13.837395 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:13.837356 2565 generic.go:356] "Generic (PLEG): container finished" podID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerID="94fdf741355c2a7abe8a7a980898ffb95dad18a93aa016b23c9a29cd8085330d" exitCode=0 Jul 16 17:56:13.837773 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:13.837427 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerDied","Data":"94fdf741355c2a7abe8a7a980898ffb95dad18a93aa016b23c9a29cd8085330d"} Jul 16 17:56:14.848194 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.848146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerStarted","Data":"b3bca3ecd1839e17b00ada8f66ebaafc4a8111068040a2b04edb0fd8c88c2133"} Jul 16 17:56:14.848699 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.848222 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerStarted","Data":"9b290ec10fda84eb1a1da49bf36f4351ab423b6a5798cca8c083ac117926909b"} Jul 16 17:56:14.848699 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.848244 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerStarted","Data":"a1d036251a8ca8d2cd0ee4017f8aebe6451b946dc678c7eab20bd877d71b4997"} Jul 16 17:56:14.848699 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.848264 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerStarted","Data":"b3194ec672da2a677e50b9f9b97f8696347e2cac50a378c7ac1cd1125a04e581"} Jul 16 17:56:14.878095 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.878055 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:14.878283 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.878135 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:14.878283 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.878159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:14.878283 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.878210 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:14.880403 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.880361 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Jul 16 17:56:14.880403 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.880390 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.41:6571: connect: connection refused" Jul 16 17:56:14.880635 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.880426 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Jul 16 17:56:14.880635 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.880471 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.41:8816: connect: connection refused" Jul 16 17:56:14.889688 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:14.889625 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podStartSLOduration=13.889603045 podStartE2EDuration="13.889603045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:56:14.887286083 +0000 UTC m=+1555.558756289" watchObservedRunningTime="2026-07-16 17:56:14.889603045 +0000 UTC m=+1555.561073247" Jul 16 17:56:17.877888 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:17.877830 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.41:6571: connect: connection refused" Jul 16 17:56:17.878403 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:17.877830 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.41:8816: connect: connection refused" Jul 16 17:56:17.878403 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:17.878107 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Jul 16 17:56:17.878403 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:17.878349 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Jul 16 17:56:20.878181 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.878143 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.878594 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.878283 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.878671 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.878653 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.878731 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.878695 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.878731 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.878713 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.878731 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.878729 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.879007 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.878986 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.879080 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.879057 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.879230 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.879216 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.883485 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.883465 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.884877 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.884860 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:56:20.887591 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:56:20.887576 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:57:22.831277 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:22.831205 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l"] Jul 16 17:57:22.831795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:22.831422 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="registry" containerID="cri-o://b3194ec672da2a677e50b9f9b97f8696347e2cac50a378c7ac1cd1125a04e581" gracePeriod=30 Jul 16 17:57:22.831795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:22.831457 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="ui" containerID="cri-o://b3bca3ecd1839e17b00ada8f66ebaafc4a8111068040a2b04edb0fd8c88c2133" gracePeriod=30 Jul 16 17:57:22.831795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:22.831494 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="offline" containerID="cri-o://9b290ec10fda84eb1a1da49bf36f4351ab423b6a5798cca8c083ac117926909b" gracePeriod=30 Jul 16 17:57:22.831795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:22.831540 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="online" containerID="cri-o://a1d036251a8ca8d2cd0ee4017f8aebe6451b946dc678c7eab20bd877d71b4997" gracePeriod=30 Jul 16 17:57:23.292031 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.291937 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp"] Jul 16 17:57:23.295863 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.295817 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.299565 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.299534 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-997h5\"" Jul 16 17:57:23.313502 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.313473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.313650 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.313569 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e323fd-0967-4811-b6de-4d2df1de53c6-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.313650 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.313595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mm2s\" (UniqueName: \"kubernetes.io/projected/e8e323fd-0967-4811-b6de-4d2df1de53c6-kube-api-access-5mm2s\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.313650 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.313633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.313828 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.313706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.313828 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.313752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.322439 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.322412 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp"] Jul 16 17:57:23.414779 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.414741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.414996 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.414799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.414996 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.414874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e323fd-0967-4811-b6de-4d2df1de53c6-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.414996 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.414898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mm2s\" (UniqueName: \"kubernetes.io/projected/e8e323fd-0967-4811-b6de-4d2df1de53c6-kube-api-access-5mm2s\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.414996 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:57:23.414930 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 16 17:57:23.414996 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:57:23.414933 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 16 17:57:23.414996 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.414933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.414996 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:57:23.414996 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-online-tls podName:e8e323fd-0967-4811-b6de-4d2df1de53c6 nodeName:}" failed. No retries permitted until 2026-07-16 17:57:23.914976272 +0000 UTC m=+1624.586446471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-online-tls") pod "feast-simple-feast-setup-796bc5bb56-6q4sp" (UID: "e8e323fd-0967-4811-b6de-4d2df1de53c6") : secret "feast-simple-feast-setup-online-tls" not found Jul 16 17:57:23.415295 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:57:23.415051 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-registry-tls podName:e8e323fd-0967-4811-b6de-4d2df1de53c6 nodeName:}" failed. No retries permitted until 2026-07-16 17:57:23.915013057 +0000 UTC m=+1624.586483242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-registry-tls") pod "feast-simple-feast-setup-796bc5bb56-6q4sp" (UID: "e8e323fd-0967-4811-b6de-4d2df1de53c6") : secret "feast-simple-feast-setup-registry-tls" not found Jul 16 17:57:23.415295 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.415123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.415295 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:57:23.415227 2565 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 16 17:57:23.415419 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.415304 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e323fd-0967-4811-b6de-4d2df1de53c6-feast-data\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.415419 ip-10-0-131-83 kubenswrapper[2565]: E0716 17:57:23.415311 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-ui-tls podName:e8e323fd-0967-4811-b6de-4d2df1de53c6 nodeName:}" failed. No retries permitted until 2026-07-16 17:57:23.915295327 +0000 UTC m=+1624.586765536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-ui-tls") pod "feast-simple-feast-setup-796bc5bb56-6q4sp" (UID: "e8e323fd-0967-4811-b6de-4d2df1de53c6") : secret "feast-simple-feast-setup-ui-tls" not found Jul 16 17:57:23.417679 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.417658 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-offline-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.431358 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.431324 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mm2s\" (UniqueName: \"kubernetes.io/projected/e8e323fd-0967-4811-b6de-4d2df1de53c6-kube-api-access-5mm2s\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.920325 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.920234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.920325 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.920288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.920795 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.920345 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.922864 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.922816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-ui-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.922996 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.922980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-online-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:23.923121 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:23.923100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-registry-tls\") pod \"feast-simple-feast-setup-796bc5bb56-6q4sp\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:24.140001 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:24.139958 2565 generic.go:356] "Generic (PLEG): container finished" podID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerID="b3bca3ecd1839e17b00ada8f66ebaafc4a8111068040a2b04edb0fd8c88c2133" exitCode=0 Jul 16 17:57:24.140001 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:24.140001 2565 generic.go:356] "Generic (PLEG): container finished" podID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerID="a1d036251a8ca8d2cd0ee4017f8aebe6451b946dc678c7eab20bd877d71b4997" exitCode=0 Jul 16 17:57:24.140228 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:24.140011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerDied","Data":"b3bca3ecd1839e17b00ada8f66ebaafc4a8111068040a2b04edb0fd8c88c2133"} Jul 16 17:57:24.140228 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:24.140054 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerDied","Data":"a1d036251a8ca8d2cd0ee4017f8aebe6451b946dc678c7eab20bd877d71b4997"} Jul 16 17:57:24.205609 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:24.205519 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:24.356063 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:24.356037 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp"] Jul 16 17:57:24.356376 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:57:24.356353 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8e323fd_0967_4811_b6de_4d2df1de53c6.slice/crio-6c0d4709db657e3723f812d3e8f1dc23c1240b19f618fb88ba978864d532e2ab WatchSource:0}: Error finding container 6c0d4709db657e3723f812d3e8f1dc23c1240b19f618fb88ba978864d532e2ab: Status 404 returned error can't find the container with id 6c0d4709db657e3723f812d3e8f1dc23c1240b19f618fb88ba978864d532e2ab Jul 16 17:57:25.146442 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:25.146406 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerStarted","Data":"4b4988cb02fc1f2496c4f6d399886e20adfd8c8f39e124ee3c56707b74a729b4"} Jul 16 17:57:25.146442 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:25.146450 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerStarted","Data":"6c0d4709db657e3723f812d3e8f1dc23c1240b19f618fb88ba978864d532e2ab"} Jul 16 17:57:30.170854 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:30.170816 2565 generic.go:356] "Generic (PLEG): container finished" podID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerID="4b4988cb02fc1f2496c4f6d399886e20adfd8c8f39e124ee3c56707b74a729b4" exitCode=0 Jul 16 17:57:30.171248 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:30.170891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerDied","Data":"4b4988cb02fc1f2496c4f6d399886e20adfd8c8f39e124ee3c56707b74a729b4"} Jul 16 17:57:30.879588 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:30.879541 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Jul 16 17:57:30.885543 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:30.885503 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Jul 16 17:57:31.177400 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:31.177316 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerStarted","Data":"c0f1b8cd62e9ac2ebfc9cf252c407ba71eebfd529ef2760c5664d53b01321816"} Jul 16 17:57:36.199346 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:36.199258 2565 generic.go:356] "Generic (PLEG): container finished" podID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerID="c0f1b8cd62e9ac2ebfc9cf252c407ba71eebfd529ef2760c5664d53b01321816" exitCode=0 Jul 16 17:57:36.199346 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:36.199298 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerDied","Data":"c0f1b8cd62e9ac2ebfc9cf252c407ba71eebfd529ef2760c5664d53b01321816"} Jul 16 17:57:37.209782 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:37.209738 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerStarted","Data":"898ad9e40facbcd96494568cccc81b5fca9bcfacc2f0b16d6af5bf4ac614cf0a"} Jul 16 17:57:37.210287 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:37.209799 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerStarted","Data":"d430e7d8e3adaa9dcba009ce12669eb6d5e913b69e6931c2685af0913b20ddb2"} Jul 16 17:57:37.210287 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:37.209820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerStarted","Data":"18d2d2c8080c9aa8e3ae4f904f32323dda973e4dc011c6f55485737491bf9b95"} Jul 16 17:57:37.210287 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:37.209857 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerStarted","Data":"6585ad2b6d0debe5c4524837d08e9d9c25e8725006e0eefe2ab2d135cc41c21b"} Jul 16 17:57:37.240732 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:37.240668 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podStartSLOduration=14.240640969 podStartE2EDuration="14.240640969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:37.239148237 +0000 UTC m=+1637.910618443" watchObservedRunningTime="2026-07-16 17:57:37.240640969 +0000 UTC m=+1637.912111174" Jul 16 17:57:39.206144 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:39.206097 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:39.206643 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:39.206166 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:39.206643 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:39.206185 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:39.206643 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:39.206203 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:39.208334 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:39.208189 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Jul 16 17:57:39.208334 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:39.208222 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.42:8816: connect: connection refused" Jul 16 17:57:39.208334 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:39.208238 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused" Jul 16 17:57:39.208334 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:39.208313 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.42:6571: connect: connection refused" Jul 16 17:57:40.879363 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:40.879315 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Jul 16 17:57:40.886067 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:40.886021 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Jul 16 17:57:42.206083 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.206038 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Jul 16 17:57:42.206967 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.206944 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:42.207075 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.207062 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:42.207454 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.207428 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:42.207590 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.207569 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:42.207780 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.207761 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:42.208006 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.207987 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:42.229446 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.229418 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:42.240460 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.240430 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:42.243501 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:42.243477 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:44.206528 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:44.206484 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:45.206078 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:45.206046 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:45.253864 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:45.253820 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 17:57:50.879590 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:50.879538 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Jul 16 17:57:50.880068 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:50.879704 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:57:50.885346 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:50.885320 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Jul 16 17:57:50.885429 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:50.885418 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:57:53.288758 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.288672 2565 generic.go:356] "Generic (PLEG): container finished" podID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerID="9b290ec10fda84eb1a1da49bf36f4351ab423b6a5798cca8c083ac117926909b" exitCode=137 Jul 16 17:57:53.288758 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.288698 2565 generic.go:356] "Generic (PLEG): container finished" podID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerID="b3194ec672da2a677e50b9f9b97f8696347e2cac50a378c7ac1cd1125a04e581" exitCode=137 Jul 16 17:57:53.288758 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.288742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerDied","Data":"9b290ec10fda84eb1a1da49bf36f4351ab423b6a5798cca8c083ac117926909b"} Jul 16 17:57:53.289215 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.288786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerDied","Data":"b3194ec672da2a677e50b9f9b97f8696347e2cac50a378c7ac1cd1125a04e581"} Jul 16 17:57:53.487945 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.487922 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:57:53.588320 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.588270 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlmzp\" (UniqueName: \"kubernetes.io/projected/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-kube-api-access-hlmzp\") pod \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " Jul 16 17:57:53.588618 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.588343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-ui-tls\") pod \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " Jul 16 17:57:53.588618 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.588363 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-online-tls\") pod \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " Jul 16 17:57:53.588618 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.588412 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-offline-tls\") pod \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " Jul 16 17:57:53.588618 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.588442 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-registry-tls\") pod \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " Jul 16 17:57:53.588618 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.588481 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-feast-data\") pod \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\" (UID: \"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d\") " Jul 16 17:57:53.589332 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.589299 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-feast-data" (OuterVolumeSpecName: "feast-data") pod "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" (UID: "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:57:53.590871 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.590819 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" (UID: "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:53.590959 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.590894 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" (UID: "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:53.590959 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.590905 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" (UID: "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:53.591072 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.591050 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-kube-api-access-hlmzp" (OuterVolumeSpecName: "kube-api-access-hlmzp") pod "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" (UID: "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d"). InnerVolumeSpecName "kube-api-access-hlmzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:57:53.591912 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.591895 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-online-tls" (OuterVolumeSpecName: "online-tls") pod "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" (UID: "a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:53.689910 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.689864 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:57:53.689910 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.689904 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:57:53.689910 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.689913 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlmzp\" (UniqueName: \"kubernetes.io/projected/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-kube-api-access-hlmzp\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:57:53.689910 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.689922 2565 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-ui-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:57:53.690185 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.689931 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:57:53.690185 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:53.689940 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 17:57:54.296057 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.295973 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" event={"ID":"a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d","Type":"ContainerDied","Data":"20fb3f9b43c0ffedfdd0f573f39428cd049d99b54a05aac1481c12f97670837e"} Jul 16 17:57:54.296057 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.296019 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l" Jul 16 17:57:54.296057 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.296031 2565 scope.go:117] "RemoveContainer" containerID="b3bca3ecd1839e17b00ada8f66ebaafc4a8111068040a2b04edb0fd8c88c2133" Jul 16 17:57:54.304724 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.304711 2565 scope.go:117] "RemoveContainer" containerID="9b290ec10fda84eb1a1da49bf36f4351ab423b6a5798cca8c083ac117926909b" Jul 16 17:57:54.312489 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.312470 2565 scope.go:117] "RemoveContainer" containerID="a1d036251a8ca8d2cd0ee4017f8aebe6451b946dc678c7eab20bd877d71b4997" Jul 16 17:57:54.318989 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.318967 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l"] Jul 16 17:57:54.320633 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.320603 2565 scope.go:117] "RemoveContainer" containerID="b3194ec672da2a677e50b9f9b97f8696347e2cac50a378c7ac1cd1125a04e581" Jul 16 17:57:54.325717 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.325693 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-n6b9l"] Jul 16 17:57:54.328596 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.328580 2565 scope.go:117] "RemoveContainer" containerID="94fdf741355c2a7abe8a7a980898ffb95dad18a93aa016b23c9a29cd8085330d" Jul 16 17:57:54.335783 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:54.335762 2565 scope.go:117] "RemoveContainer" containerID="63d0a3f17b8709d28a045123ce57ba4e2e71d9908822cda25a9efd0a5c362fc1" Jul 16 17:57:55.876855 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:57:55.876806 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" path="/var/lib/kubelet/pods/a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d/volumes" Jul 16 17:58:46.692736 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.692651 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp"] Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693007 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="registry" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693020 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="registry" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693032 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="feast-apply" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693038 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="feast-apply" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693048 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="feast-init" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693053 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="feast-init" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693062 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="online" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693067 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="online" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693075 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="offline" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693080 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="offline" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693088 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="ui" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693093 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="ui" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693140 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="offline" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693147 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="online" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693154 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="ui" Jul 16 17:58:46.693236 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.693161 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27e21aa-a60c-41f1-a7ba-3e7ed214ad8d" containerName="registry" Jul 16 17:58:46.697894 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.697874 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.703755 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.703732 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-5rr7m\"" Jul 16 17:58:46.703943 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.703902 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 16 17:58:46.703998 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.703959 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 16 17:58:46.704707 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.704472 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 16 17:58:46.704707 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.704495 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 16 17:58:46.704707 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.704519 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 16 17:58:46.704707 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.704472 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 16 17:58:46.719876 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.719823 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp"] Jul 16 17:58:46.849119 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.849091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-ui-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.849314 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.849129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-offline-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.849314 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.849148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-online-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.849314 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.849196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcjd6\" (UniqueName: \"kubernetes.io/projected/9d458be0-11f0-4668-a040-e1644e4ad68e-kube-api-access-lcjd6\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.849314 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.849253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9d458be0-11f0-4668-a040-e1644e4ad68e-registry-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.849314 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.849298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d458be0-11f0-4668-a040-e1644e4ad68e-feast-data\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.950292 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.950202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcjd6\" (UniqueName: \"kubernetes.io/projected/9d458be0-11f0-4668-a040-e1644e4ad68e-kube-api-access-lcjd6\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.950292 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.950252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9d458be0-11f0-4668-a040-e1644e4ad68e-registry-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.950292 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.950277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d458be0-11f0-4668-a040-e1644e4ad68e-feast-data\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.950558 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.950314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-ui-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.950558 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.950338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-offline-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.950558 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.950354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-online-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.950801 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.950777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d458be0-11f0-4668-a040-e1644e4ad68e-feast-data\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.951039 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.951007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9d458be0-11f0-4668-a040-e1644e4ad68e-registry-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.952900 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.952876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-ui-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.953284 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.953266 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-online-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.953445 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.953422 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-offline-tls\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:46.961275 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:46.961252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcjd6\" (UniqueName: \"kubernetes.io/projected/9d458be0-11f0-4668-a040-e1644e4ad68e-kube-api-access-lcjd6\") pod \"feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:47.006766 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:47.006733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:58:47.153525 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:47.153475 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp"] Jul 16 17:58:47.159545 ip-10-0-131-83 kubenswrapper[2565]: W0716 17:58:47.159516 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d458be0_11f0_4668_a040_e1644e4ad68e.slice/crio-e6b2acec4d1c8014c870f299f872715a153cc11c6e0843fefc637cb34db3d0f2 WatchSource:0}: Error finding container e6b2acec4d1c8014c870f299f872715a153cc11c6e0843fefc637cb34db3d0f2: Status 404 returned error can't find the container with id e6b2acec4d1c8014c870f299f872715a153cc11c6e0843fefc637cb34db3d0f2 Jul 16 17:58:47.518558 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:47.518458 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerStarted","Data":"706d0b49d312ccbc1279800623150a63f2d5db45aa5172d66a91b14e8a41b52d"} Jul 16 17:58:47.518558 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:47.518509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerStarted","Data":"e6b2acec4d1c8014c870f299f872715a153cc11c6e0843fefc637cb34db3d0f2"} Jul 16 17:58:52.542274 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:52.542242 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerID="706d0b49d312ccbc1279800623150a63f2d5db45aa5172d66a91b14e8a41b52d" exitCode=0 Jul 16 17:58:52.542761 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:52.542299 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerDied","Data":"706d0b49d312ccbc1279800623150a63f2d5db45aa5172d66a91b14e8a41b52d"} Jul 16 17:58:53.548274 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:53.548238 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerStarted","Data":"d0159f1dcaf63edf7e3f9644c51443e5f451451e6e9eeebfd25f8e24455c171d"} Jul 16 17:58:58.573151 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:58.573115 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerID="d0159f1dcaf63edf7e3f9644c51443e5f451451e6e9eeebfd25f8e24455c171d" exitCode=0 Jul 16 17:58:58.573565 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:58.573165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerDied","Data":"d0159f1dcaf63edf7e3f9644c51443e5f451451e6e9eeebfd25f8e24455c171d"} Jul 16 17:58:59.581621 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:59.581580 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerStarted","Data":"f36f508f19eb642e3786d7984ac72e0a386eb3041eb4adf392abef657afd9c4a"} Jul 16 17:58:59.582032 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:59.581642 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerStarted","Data":"da65ab007f5692d6a302bf87ef6d29c402ba4c12fb24e6c32a7145fd65cc42c4"} Jul 16 17:58:59.582032 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:59.581662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerStarted","Data":"2a999c990489f481d93ce481de5752a4a51cf7f5d2963bb813499a4bd899178a"} Jul 16 17:58:59.621787 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:58:59.612732 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podStartSLOduration=13.612706654 podStartE2EDuration="13.612706654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:58:59.611117054 +0000 UTC m=+1720.282587259" watchObservedRunningTime="2026-07-16 17:58:59.612706654 +0000 UTC m=+1720.284176852" Jul 16 17:59:02.006925 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:02.006883 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:02.007461 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:02.006947 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:02.007461 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:02.006968 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:02.008561 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:02.008525 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.43:8443: connect: connection refused" Jul 16 17:59:02.008705 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:02.008559 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.43:8816: connect: connection refused" Jul 16 17:59:02.008705 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:02.008571 2565 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.43:6567/health\": dial tcp 10.133.0.43:6567: connect: connection refused" Jul 16 17:59:05.008038 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:05.008011 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:05.008527 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:05.008151 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:05.008618 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:05.008591 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:05.008688 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:05.008640 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:05.008914 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:05.008894 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:05.009229 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:05.009215 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:05.014515 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:05.014491 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:05.610905 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:05.610872 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 17:59:05.615688 ip-10-0-131-83 kubenswrapper[2565]: I0716 17:59:05.615665 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 18:00:07.721812 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:07.721778 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp"] Jul 16 18:00:07.722358 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:07.722042 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="online" containerID="cri-o://2a999c990489f481d93ce481de5752a4a51cf7f5d2963bb813499a4bd899178a" gracePeriod=30 Jul 16 18:00:07.722358 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:07.722070 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="offline" containerID="cri-o://da65ab007f5692d6a302bf87ef6d29c402ba4c12fb24e6c32a7145fd65cc42c4" gracePeriod=30 Jul 16 18:00:07.722358 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:07.722135 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="ui" containerID="cri-o://f36f508f19eb642e3786d7984ac72e0a386eb3041eb4adf392abef657afd9c4a" gracePeriod=30 Jul 16 18:00:07.927401 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:07.927366 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp"] Jul 16 18:00:07.927640 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:07.927593 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="registry" containerID="cri-o://6585ad2b6d0debe5c4524837d08e9d9c25e8725006e0eefe2ab2d135cc41c21b" gracePeriod=30 Jul 16 18:00:07.927708 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:07.927668 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="ui" containerID="cri-o://898ad9e40facbcd96494568cccc81b5fca9bcfacc2f0b16d6af5bf4ac614cf0a" gracePeriod=30 Jul 16 18:00:07.927708 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:07.927676 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="offline" containerID="cri-o://d430e7d8e3adaa9dcba009ce12669eb6d5e913b69e6931c2685af0913b20ddb2" gracePeriod=30 Jul 16 18:00:07.927822 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:07.927666 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="online" containerID="cri-o://18d2d2c8080c9aa8e3ae4f904f32323dda973e4dc011c6f55485737491bf9b95" gracePeriod=30 Jul 16 18:00:08.884326 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:08.884288 2565 generic.go:356] "Generic (PLEG): container finished" podID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerID="898ad9e40facbcd96494568cccc81b5fca9bcfacc2f0b16d6af5bf4ac614cf0a" exitCode=0 Jul 16 18:00:08.884326 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:08.884322 2565 generic.go:356] "Generic (PLEG): container finished" podID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerID="18d2d2c8080c9aa8e3ae4f904f32323dda973e4dc011c6f55485737491bf9b95" exitCode=0 Jul 16 18:00:08.884830 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:08.884451 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerDied","Data":"898ad9e40facbcd96494568cccc81b5fca9bcfacc2f0b16d6af5bf4ac614cf0a"} Jul 16 18:00:08.884830 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:08.884492 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerDied","Data":"18d2d2c8080c9aa8e3ae4f904f32323dda973e4dc011c6f55485737491bf9b95"} Jul 16 18:00:08.887540 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:08.887518 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerID="f36f508f19eb642e3786d7984ac72e0a386eb3041eb4adf392abef657afd9c4a" exitCode=0 Jul 16 18:00:08.887540 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:08.887536 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerID="2a999c990489f481d93ce481de5752a4a51cf7f5d2963bb813499a4bd899178a" exitCode=0 Jul 16 18:00:08.887708 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:08.887566 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerDied","Data":"f36f508f19eb642e3786d7984ac72e0a386eb3041eb4adf392abef657afd9c4a"} Jul 16 18:00:08.887708 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:08.887590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerDied","Data":"2a999c990489f481d93ce481de5752a4a51cf7f5d2963bb813499a4bd899178a"} Jul 16 18:00:12.241005 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:12.240904 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused" Jul 16 18:00:15.009200 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:15.009146 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.43:8443: connect: connection refused" Jul 16 18:00:15.253288 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:15.253223 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Jul 16 18:00:15.611432 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:15.611389 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.43:6567/health\": dial tcp 10.133.0.43:6567: connect: connection refused" Jul 16 18:00:22.241046 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:22.241001 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused" Jul 16 18:00:25.009631 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:25.009576 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.43:8443: connect: connection refused" Jul 16 18:00:25.253828 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:25.253785 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Jul 16 18:00:25.611231 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:25.611185 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.43:6567/health\": dial tcp 10.133.0.43:6567: connect: connection refused" Jul 16 18:00:32.241304 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:32.241265 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused" Jul 16 18:00:32.241780 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:32.241389 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 18:00:35.009654 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:35.009604 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.43:8443: connect: connection refused" Jul 16 18:00:35.010102 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:35.009767 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 18:00:35.254314 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:35.254272 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Jul 16 18:00:35.254485 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:35.254400 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 18:00:35.612046 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:35.612004 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.43:6567/health\": dial tcp 10.133.0.43:6567: connect: connection refused" Jul 16 18:00:35.612259 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:35.612129 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 18:00:38.008192 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.008106 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerID="da65ab007f5692d6a302bf87ef6d29c402ba4c12fb24e6c32a7145fd65cc42c4" exitCode=137 Jul 16 18:00:38.008192 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.008170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerDied","Data":"da65ab007f5692d6a302bf87ef6d29c402ba4c12fb24e6c32a7145fd65cc42c4"} Jul 16 18:00:38.011177 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.011151 2565 generic.go:356] "Generic (PLEG): container finished" podID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerID="d430e7d8e3adaa9dcba009ce12669eb6d5e913b69e6931c2685af0913b20ddb2" exitCode=137 Jul 16 18:00:38.011177 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.011176 2565 generic.go:356] "Generic (PLEG): container finished" podID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerID="6585ad2b6d0debe5c4524837d08e9d9c25e8725006e0eefe2ab2d135cc41c21b" exitCode=137 Jul 16 18:00:38.011321 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.011228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerDied","Data":"d430e7d8e3adaa9dcba009ce12669eb6d5e913b69e6931c2685af0913b20ddb2"} Jul 16 18:00:38.011321 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.011259 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerDied","Data":"6585ad2b6d0debe5c4524837d08e9d9c25e8725006e0eefe2ab2d135cc41c21b"} Jul 16 18:00:38.374996 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.374967 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 18:00:38.414857 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.414817 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-ui-tls\") pod \"9d458be0-11f0-4668-a040-e1644e4ad68e\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " Jul 16 18:00:38.415003 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.414871 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-online-tls\") pod \"9d458be0-11f0-4668-a040-e1644e4ad68e\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " Jul 16 18:00:38.415003 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.414895 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-offline-tls\") pod \"9d458be0-11f0-4668-a040-e1644e4ad68e\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " Jul 16 18:00:38.415087 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.415065 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d458be0-11f0-4668-a040-e1644e4ad68e-feast-data\") pod \"9d458be0-11f0-4668-a040-e1644e4ad68e\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " Jul 16 18:00:38.415154 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.415137 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcjd6\" (UniqueName: \"kubernetes.io/projected/9d458be0-11f0-4668-a040-e1644e4ad68e-kube-api-access-lcjd6\") pod \"9d458be0-11f0-4668-a040-e1644e4ad68e\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " Jul 16 18:00:38.415201 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.415178 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9d458be0-11f0-4668-a040-e1644e4ad68e-registry-tls\") pod \"9d458be0-11f0-4668-a040-e1644e4ad68e\" (UID: \"9d458be0-11f0-4668-a040-e1644e4ad68e\") " Jul 16 18:00:38.415636 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.415606 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d458be0-11f0-4668-a040-e1644e4ad68e-feast-data" (OuterVolumeSpecName: "feast-data") pod "9d458be0-11f0-4668-a040-e1644e4ad68e" (UID: "9d458be0-11f0-4668-a040-e1644e4ad68e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:00:38.415995 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.415915 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d458be0-11f0-4668-a040-e1644e4ad68e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9d458be0-11f0-4668-a040-e1644e4ad68e" (UID: "9d458be0-11f0-4668-a040-e1644e4ad68e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:00:38.417379 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.417354 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "9d458be0-11f0-4668-a040-e1644e4ad68e" (UID: "9d458be0-11f0-4668-a040-e1644e4ad68e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:00:38.417464 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.417385 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9d458be0-11f0-4668-a040-e1644e4ad68e" (UID: "9d458be0-11f0-4668-a040-e1644e4ad68e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:00:38.417464 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.417402 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d458be0-11f0-4668-a040-e1644e4ad68e-kube-api-access-lcjd6" (OuterVolumeSpecName: "kube-api-access-lcjd6") pod "9d458be0-11f0-4668-a040-e1644e4ad68e" (UID: "9d458be0-11f0-4668-a040-e1644e4ad68e"). InnerVolumeSpecName "kube-api-access-lcjd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:00:38.417867 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.417830 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-online-tls" (OuterVolumeSpecName: "online-tls") pod "9d458be0-11f0-4668-a040-e1644e4ad68e" (UID: "9d458be0-11f0-4668-a040-e1644e4ad68e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:00:38.516606 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.516576 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9d458be0-11f0-4668-a040-e1644e4ad68e-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.516606 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.516606 2565 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-ui-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.516790 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.516636 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.516790 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.516644 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9d458be0-11f0-4668-a040-e1644e4ad68e-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.516790 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.516652 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9d458be0-11f0-4668-a040-e1644e4ad68e-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.516790 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.516659 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcjd6\" (UniqueName: \"kubernetes.io/projected/9d458be0-11f0-4668-a040-e1644e4ad68e-kube-api-access-lcjd6\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.562094 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.562071 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 18:00:38.617784 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.617750 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e323fd-0967-4811-b6de-4d2df1de53c6-feast-data\") pod \"e8e323fd-0967-4811-b6de-4d2df1de53c6\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " Jul 16 18:00:38.617784 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.617786 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mm2s\" (UniqueName: \"kubernetes.io/projected/e8e323fd-0967-4811-b6de-4d2df1de53c6-kube-api-access-5mm2s\") pod \"e8e323fd-0967-4811-b6de-4d2df1de53c6\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " Jul 16 18:00:38.618004 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.617812 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-online-tls\") pod \"e8e323fd-0967-4811-b6de-4d2df1de53c6\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " Jul 16 18:00:38.618004 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.617862 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-offline-tls\") pod \"e8e323fd-0967-4811-b6de-4d2df1de53c6\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " Jul 16 18:00:38.618004 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.617898 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-ui-tls\") pod \"e8e323fd-0967-4811-b6de-4d2df1de53c6\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " Jul 16 18:00:38.618004 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.617951 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-registry-tls\") pod \"e8e323fd-0967-4811-b6de-4d2df1de53c6\" (UID: \"e8e323fd-0967-4811-b6de-4d2df1de53c6\") " Jul 16 18:00:38.618369 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.618342 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e323fd-0967-4811-b6de-4d2df1de53c6-feast-data" (OuterVolumeSpecName: "feast-data") pod "e8e323fd-0967-4811-b6de-4d2df1de53c6" (UID: "e8e323fd-0967-4811-b6de-4d2df1de53c6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:00:38.620290 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.620269 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e8e323fd-0967-4811-b6de-4d2df1de53c6" (UID: "e8e323fd-0967-4811-b6de-4d2df1de53c6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:00:38.620587 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.620568 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e8e323fd-0967-4811-b6de-4d2df1de53c6" (UID: "e8e323fd-0967-4811-b6de-4d2df1de53c6"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:00:38.620646 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.620595 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e323fd-0967-4811-b6de-4d2df1de53c6-kube-api-access-5mm2s" (OuterVolumeSpecName: "kube-api-access-5mm2s") pod "e8e323fd-0967-4811-b6de-4d2df1de53c6" (UID: "e8e323fd-0967-4811-b6de-4d2df1de53c6"). InnerVolumeSpecName "kube-api-access-5mm2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:00:38.620646 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.620635 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-online-tls" (OuterVolumeSpecName: "online-tls") pod "e8e323fd-0967-4811-b6de-4d2df1de53c6" (UID: "e8e323fd-0967-4811-b6de-4d2df1de53c6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:00:38.620713 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.620651 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e8e323fd-0967-4811-b6de-4d2df1de53c6" (UID: "e8e323fd-0967-4811-b6de-4d2df1de53c6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:00:38.718986 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.718952 2565 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-offline-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.718986 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.718980 2565 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-ui-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.718986 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.718988 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.718986 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.718997 2565 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e323fd-0967-4811-b6de-4d2df1de53c6-feast-data\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.719236 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.719006 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mm2s\" (UniqueName: \"kubernetes.io/projected/e8e323fd-0967-4811-b6de-4d2df1de53c6-kube-api-access-5mm2s\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:38.719236 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:38.719014 2565 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e8e323fd-0967-4811-b6de-4d2df1de53c6-online-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:00:39.018118 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.018012 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" Jul 16 18:00:39.018118 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.018017 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp" event={"ID":"9d458be0-11f0-4668-a040-e1644e4ad68e","Type":"ContainerDied","Data":"e6b2acec4d1c8014c870f299f872715a153cc11c6e0843fefc637cb34db3d0f2"} Jul 16 18:00:39.018645 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.018160 2565 scope.go:117] "RemoveContainer" containerID="f36f508f19eb642e3786d7984ac72e0a386eb3041eb4adf392abef657afd9c4a" Jul 16 18:00:39.022066 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.022035 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" event={"ID":"e8e323fd-0967-4811-b6de-4d2df1de53c6","Type":"ContainerDied","Data":"6c0d4709db657e3723f812d3e8f1dc23c1240b19f618fb88ba978864d532e2ab"} Jul 16 18:00:39.022217 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.022137 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp" Jul 16 18:00:39.029989 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.029972 2565 scope.go:117] "RemoveContainer" containerID="da65ab007f5692d6a302bf87ef6d29c402ba4c12fb24e6c32a7145fd65cc42c4" Jul 16 18:00:39.038256 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.038238 2565 scope.go:117] "RemoveContainer" containerID="2a999c990489f481d93ce481de5752a4a51cf7f5d2963bb813499a4bd899178a" Jul 16 18:00:39.045281 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.045256 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp"] Jul 16 18:00:39.046332 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.046312 2565 scope.go:117] "RemoveContainer" containerID="d0159f1dcaf63edf7e3f9644c51443e5f451451e6e9eeebfd25f8e24455c171d" Jul 16 18:00:39.050364 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.050340 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74bdf4fbf8-kpwgp"] Jul 16 18:00:39.054970 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.054947 2565 scope.go:117] "RemoveContainer" containerID="706d0b49d312ccbc1279800623150a63f2d5db45aa5172d66a91b14e8a41b52d" Jul 16 18:00:39.062757 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.062738 2565 scope.go:117] "RemoveContainer" containerID="898ad9e40facbcd96494568cccc81b5fca9bcfacc2f0b16d6af5bf4ac614cf0a" Jul 16 18:00:39.070432 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.070411 2565 scope.go:117] "RemoveContainer" containerID="d430e7d8e3adaa9dcba009ce12669eb6d5e913b69e6931c2685af0913b20ddb2" Jul 16 18:00:39.070537 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.070523 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp"] Jul 16 18:00:39.075882 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.075862 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-796bc5bb56-6q4sp"] Jul 16 18:00:39.078038 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.078021 2565 scope.go:117] "RemoveContainer" containerID="18d2d2c8080c9aa8e3ae4f904f32323dda973e4dc011c6f55485737491bf9b95" Jul 16 18:00:39.085228 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.085214 2565 scope.go:117] "RemoveContainer" containerID="6585ad2b6d0debe5c4524837d08e9d9c25e8725006e0eefe2ab2d135cc41c21b" Jul 16 18:00:39.092388 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.092373 2565 scope.go:117] "RemoveContainer" containerID="c0f1b8cd62e9ac2ebfc9cf252c407ba71eebfd529ef2760c5664d53b01321816" Jul 16 18:00:39.099206 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.099191 2565 scope.go:117] "RemoveContainer" containerID="4b4988cb02fc1f2496c4f6d399886e20adfd8c8f39e124ee3c56707b74a729b4" Jul 16 18:00:39.876441 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.876408 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" path="/var/lib/kubelet/pods/9d458be0-11f0-4668-a040-e1644e4ad68e/volumes" Jul 16 18:00:39.876959 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:39.876946 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" path="/var/lib/kubelet/pods/e8e323fd-0967-4811-b6de-4d2df1de53c6/volumes" Jul 16 18:00:58.029598 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029566 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cqz48/must-gather-9dkbw"] Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029916 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="online" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029927 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="online" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029935 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="ui" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029941 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="ui" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029947 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="feast-init" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029954 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="feast-init" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029960 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="feast-apply" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029965 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="feast-apply" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029974 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="registry" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029979 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="registry" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029989 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="online" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.029994 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="online" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030000 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="ui" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030005 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="ui" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030012 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="feast-init" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030016 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="feast-init" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030025 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="offline" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030029 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="offline" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030037 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="feast-apply" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030041 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="feast-apply" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030054 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="offline" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030059 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="offline" Jul 16 18:00:58.030098 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030105 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="online" Jul 16 18:00:58.030766 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030113 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="ui" Jul 16 18:00:58.030766 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030120 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="registry" Jul 16 18:00:58.030766 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030125 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="online" Jul 16 18:00:58.030766 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030134 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d458be0-11f0-4668-a040-e1644e4ad68e" containerName="offline" Jul 16 18:00:58.030766 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030140 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="offline" Jul 16 18:00:58.030766 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.030145 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e323fd-0967-4811-b6de-4d2df1de53c6" containerName="ui" Jul 16 18:00:58.033228 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.033210 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:00:58.036772 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.036754 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cqz48\"/\"default-dockercfg-ftwpp\"" Jul 16 18:00:58.037984 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.037959 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cqz48\"/\"openshift-service-ca.crt\"" Jul 16 18:00:58.038080 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.038019 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cqz48\"/\"kube-root-ca.crt\"" Jul 16 18:00:58.045229 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.045209 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cqz48/must-gather-9dkbw"] Jul 16 18:00:58.174433 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.174402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kthx\" (UniqueName: \"kubernetes.io/projected/6083807b-72c8-4f79-b464-c17138fd66b2-kube-api-access-7kthx\") pod \"must-gather-9dkbw\" (UID: \"6083807b-72c8-4f79-b464-c17138fd66b2\") " pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:00:58.174622 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.174496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6083807b-72c8-4f79-b464-c17138fd66b2-must-gather-output\") pod \"must-gather-9dkbw\" (UID: \"6083807b-72c8-4f79-b464-c17138fd66b2\") " pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:00:58.275706 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.275671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kthx\" (UniqueName: \"kubernetes.io/projected/6083807b-72c8-4f79-b464-c17138fd66b2-kube-api-access-7kthx\") pod \"must-gather-9dkbw\" (UID: \"6083807b-72c8-4f79-b464-c17138fd66b2\") " pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:00:58.275879 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.275765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6083807b-72c8-4f79-b464-c17138fd66b2-must-gather-output\") pod \"must-gather-9dkbw\" (UID: \"6083807b-72c8-4f79-b464-c17138fd66b2\") " pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:00:58.276077 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.276059 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6083807b-72c8-4f79-b464-c17138fd66b2-must-gather-output\") pod \"must-gather-9dkbw\" (UID: \"6083807b-72c8-4f79-b464-c17138fd66b2\") " pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:00:58.289583 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.289528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kthx\" (UniqueName: \"kubernetes.io/projected/6083807b-72c8-4f79-b464-c17138fd66b2-kube-api-access-7kthx\") pod \"must-gather-9dkbw\" (UID: \"6083807b-72c8-4f79-b464-c17138fd66b2\") " pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:00:58.354320 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.354285 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:00:58.488153 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.488131 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cqz48/must-gather-9dkbw"] Jul 16 18:00:58.490292 ip-10-0-131-83 kubenswrapper[2565]: W0716 18:00:58.490259 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6083807b_72c8_4f79_b464_c17138fd66b2.slice/crio-427b75640331ad9ceb04506e2eb712dc048573f0b69c3ef6789cce5597200f95 WatchSource:0}: Error finding container 427b75640331ad9ceb04506e2eb712dc048573f0b69c3ef6789cce5597200f95: Status 404 returned error can't find the container with id 427b75640331ad9ceb04506e2eb712dc048573f0b69c3ef6789cce5597200f95 Jul 16 18:00:58.492021 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:58.492005 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 18:00:59.105820 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:00:59.105779 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cqz48/must-gather-9dkbw" event={"ID":"6083807b-72c8-4f79-b464-c17138fd66b2","Type":"ContainerStarted","Data":"427b75640331ad9ceb04506e2eb712dc048573f0b69c3ef6789cce5597200f95"} Jul 16 18:01:05.143992 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:05.143956 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cqz48/must-gather-9dkbw" event={"ID":"6083807b-72c8-4f79-b464-c17138fd66b2","Type":"ContainerStarted","Data":"98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d"} Jul 16 18:01:06.150505 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:06.150460 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cqz48/must-gather-9dkbw" event={"ID":"6083807b-72c8-4f79-b464-c17138fd66b2","Type":"ContainerStarted","Data":"f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254"} Jul 16 18:01:06.170529 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:06.170479 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cqz48/must-gather-9dkbw" podStartSLOduration=1.670532423 podStartE2EDuration="8.170463032s" totalImagesPullingTime="6.499930609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:01:06.168210591 +0000 UTC m=+1846.839680795" watchObservedRunningTime="2026-07-16 18:01:06.170463032 +0000 UTC m=+1846.841933235" Jul 16 18:01:15.189393 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:15.189356 2565 generic.go:356] "Generic (PLEG): container finished" podID="6083807b-72c8-4f79-b464-c17138fd66b2" containerID="98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d" exitCode=0 Jul 16 18:01:15.189816 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:15.189430 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cqz48/must-gather-9dkbw" event={"ID":"6083807b-72c8-4f79-b464-c17138fd66b2","Type":"ContainerDied","Data":"98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d"} Jul 16 18:01:15.189816 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:15.189766 2565 scope.go:117] "RemoveContainer" containerID="98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d" Jul 16 18:01:15.732991 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:15.732959 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cqz48_must-gather-9dkbw_6083807b-72c8-4f79-b464-c17138fd66b2/gather/0.log" Jul 16 18:01:19.206066 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:19.206038 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dddd5_9c53a8a9-49b1-48d7-acd3-6aaf56b998ed/global-pull-secret-syncer/0.log" Jul 16 18:01:19.375538 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:19.375502 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xbth4_f0e7f5fe-bb08-43f7-8995-3767512c6544/konnectivity-agent/0.log" Jul 16 18:01:19.402623 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:19.402591 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-83.ec2.internal_e83d90178041f29950db3df0e41da00c/haproxy/0.log" Jul 16 18:01:21.135770 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.135734 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-cqz48/must-gather-9dkbw"] Jul 16 18:01:21.136180 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.135953 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-cqz48/must-gather-9dkbw" podUID="6083807b-72c8-4f79-b464-c17138fd66b2" containerName="copy" containerID="cri-o://f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254" gracePeriod=2 Jul 16 18:01:21.141886 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.141859 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-cqz48/must-gather-9dkbw"] Jul 16 18:01:21.370695 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.370675 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cqz48_must-gather-9dkbw_6083807b-72c8-4f79-b464-c17138fd66b2/copy/0.log" Jul 16 18:01:21.373217 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.373201 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:01:21.375639 ip-10-0-131-83 kubenswrapper[2565]: E0716 18:01:21.375614 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9dkbw\" is forbidden: User \"system:node:ip-10-0-131-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-cqz48\": no relationship found between node 'ip-10-0-131-83.ec2.internal' and this object" podUID="6083807b-72c8-4f79-b464-c17138fd66b2" pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:01:21.482386 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.482283 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6083807b-72c8-4f79-b464-c17138fd66b2-must-gather-output\") pod \"6083807b-72c8-4f79-b464-c17138fd66b2\" (UID: \"6083807b-72c8-4f79-b464-c17138fd66b2\") " Jul 16 18:01:21.482386 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.482372 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kthx\" (UniqueName: \"kubernetes.io/projected/6083807b-72c8-4f79-b464-c17138fd66b2-kube-api-access-7kthx\") pod \"6083807b-72c8-4f79-b464-c17138fd66b2\" (UID: \"6083807b-72c8-4f79-b464-c17138fd66b2\") " Jul 16 18:01:21.482655 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.482633 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6083807b-72c8-4f79-b464-c17138fd66b2-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "6083807b-72c8-4f79-b464-c17138fd66b2" (UID: "6083807b-72c8-4f79-b464-c17138fd66b2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:01:21.484593 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.484567 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6083807b-72c8-4f79-b464-c17138fd66b2-kube-api-access-7kthx" (OuterVolumeSpecName: "kube-api-access-7kthx") pod "6083807b-72c8-4f79-b464-c17138fd66b2" (UID: "6083807b-72c8-4f79-b464-c17138fd66b2"). InnerVolumeSpecName "kube-api-access-7kthx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:01:21.582916 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.582877 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kthx\" (UniqueName: \"kubernetes.io/projected/6083807b-72c8-4f79-b464-c17138fd66b2-kube-api-access-7kthx\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:01:21.582916 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.582912 2565 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6083807b-72c8-4f79-b464-c17138fd66b2-must-gather-output\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 16 18:01:21.876681 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:21.876645 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6083807b-72c8-4f79-b464-c17138fd66b2" path="/var/lib/kubelet/pods/6083807b-72c8-4f79-b464-c17138fd66b2/volumes" Jul 16 18:01:22.220767 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.220688 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cqz48_must-gather-9dkbw_6083807b-72c8-4f79-b464-c17138fd66b2/copy/0.log" Jul 16 18:01:22.221155 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.221039 2565 generic.go:356] "Generic (PLEG): container finished" podID="6083807b-72c8-4f79-b464-c17138fd66b2" containerID="f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254" exitCode=143 Jul 16 18:01:22.221155 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.221091 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cqz48/must-gather-9dkbw" Jul 16 18:01:22.221155 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.221137 2565 scope.go:117] "RemoveContainer" containerID="f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254" Jul 16 18:01:22.228832 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.228809 2565 scope.go:117] "RemoveContainer" containerID="98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d" Jul 16 18:01:22.240257 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.240230 2565 scope.go:117] "RemoveContainer" containerID="f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254" Jul 16 18:01:22.240546 ip-10-0-131-83 kubenswrapper[2565]: E0716 18:01:22.240527 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254\": container with ID starting with f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254 not found: ID does not exist" containerID="f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254" Jul 16 18:01:22.240589 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.240556 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254"} err="failed to get container status \"f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254\": rpc error: code = NotFound desc = could not find container \"f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254\": container with ID starting with f89db6e873c8fb9f0e4a096544698e668dd8d07bc03c278111264c97d598b254 not found: ID does not exist" Jul 16 18:01:22.240589 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.240574 2565 scope.go:117] "RemoveContainer" containerID="98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d" Jul 16 18:01:22.240827 ip-10-0-131-83 kubenswrapper[2565]: E0716 18:01:22.240808 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d\": container with ID starting with 98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d not found: ID does not exist" containerID="98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d" Jul 16 18:01:22.240919 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.240863 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d"} err="failed to get container status \"98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d\": rpc error: code = NotFound desc = could not find container \"98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d\": container with ID starting with 98803b556ca330d25edd11f4b9e824694dac61ee7830e7aa0eff6fc06a34ec0d not found: ID does not exist" Jul 16 18:01:22.776016 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.775997 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcefc0e9-6263-4dc4-8b32-7596c52ac9ab/alertmanager/0.log" Jul 16 18:01:22.830287 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.830259 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcefc0e9-6263-4dc4-8b32-7596c52ac9ab/config-reloader/0.log" Jul 16 18:01:22.891323 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.891298 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcefc0e9-6263-4dc4-8b32-7596c52ac9ab/kube-rbac-proxy-web/0.log" Jul 16 18:01:22.933891 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.933865 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcefc0e9-6263-4dc4-8b32-7596c52ac9ab/kube-rbac-proxy/0.log" Jul 16 18:01:22.976284 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:22.976254 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcefc0e9-6263-4dc4-8b32-7596c52ac9ab/kube-rbac-proxy-metric/0.log" Jul 16 18:01:23.020108 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:23.020041 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcefc0e9-6263-4dc4-8b32-7596c52ac9ab/prom-label-proxy/0.log" Jul 16 18:01:23.077884 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:23.077853 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcefc0e9-6263-4dc4-8b32-7596c52ac9ab/init-config-reloader/0.log" Jul 16 18:01:23.348379 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:23.348346 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-597f4687df-q7z9v_c3ad62a1-601f-4bb1-acc5-c567cc32f9cb/metrics-server/0.log" Jul 16 18:01:23.378731 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:23.378698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-b9n2x_14753c04-960a-41c7-8111-95101d4dfeba/monitoring-plugin/0.log" Jul 16 18:01:23.521521 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:23.521486 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qwcvg_68f611a5-8e24-4dbe-a1fd-316365337579/node-exporter/0.log" Jul 16 18:01:23.554683 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:23.554656 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qwcvg_68f611a5-8e24-4dbe-a1fd-316365337579/kube-rbac-proxy/0.log" Jul 16 18:01:23.585942 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:23.585920 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qwcvg_68f611a5-8e24-4dbe-a1fd-316365337579/init-textfile/0.log" Jul 16 18:01:24.129380 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.129354 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-rgxtq_440ed8b2-8190-4ba9-92bf-2cfc51c84e1f/prometheus-operator/0.log" Jul 16 18:01:24.164325 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.164293 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-rgxtq_440ed8b2-8190-4ba9-92bf-2cfc51c84e1f/kube-rbac-proxy/0.log" Jul 16 18:01:24.210832 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.210801 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-nj4td_6fd55a34-6d28-4693-b9d4-aef3dc141941/prometheus-operator-admission-webhook/0.log" Jul 16 18:01:24.272297 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.272274 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fccd8557-gwgwd_c06664ca-c52a-4f51-aa10-5ea66c04df7f/telemeter-client/0.log" Jul 16 18:01:24.327538 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.327509 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fccd8557-gwgwd_c06664ca-c52a-4f51-aa10-5ea66c04df7f/reload/0.log" Jul 16 18:01:24.372154 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.372127 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fccd8557-gwgwd_c06664ca-c52a-4f51-aa10-5ea66c04df7f/kube-rbac-proxy/0.log" Jul 16 18:01:24.423367 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.423280 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f5f78957b-zphtm_f3e255e5-d6e8-4fce-a1eb-1db19da83c80/thanos-query/0.log" Jul 16 18:01:24.455053 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.455022 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f5f78957b-zphtm_f3e255e5-d6e8-4fce-a1eb-1db19da83c80/kube-rbac-proxy-web/0.log" Jul 16 18:01:24.499856 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.499805 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f5f78957b-zphtm_f3e255e5-d6e8-4fce-a1eb-1db19da83c80/kube-rbac-proxy/0.log" Jul 16 18:01:24.548281 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.548254 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f5f78957b-zphtm_f3e255e5-d6e8-4fce-a1eb-1db19da83c80/prom-label-proxy/0.log" Jul 16 18:01:24.597537 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.597505 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f5f78957b-zphtm_f3e255e5-d6e8-4fce-a1eb-1db19da83c80/kube-rbac-proxy-rules/0.log" Jul 16 18:01:24.636890 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:24.636866 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f5f78957b-zphtm_f3e255e5-d6e8-4fce-a1eb-1db19da83c80/kube-rbac-proxy-metrics/0.log" Jul 16 18:01:26.715546 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.715507 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct"] Jul 16 18:01:26.715948 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.715816 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6083807b-72c8-4f79-b464-c17138fd66b2" containerName="copy" Jul 16 18:01:26.715948 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.715828 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6083807b-72c8-4f79-b464-c17138fd66b2" containerName="copy" Jul 16 18:01:26.715948 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.715852 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6083807b-72c8-4f79-b464-c17138fd66b2" containerName="gather" Jul 16 18:01:26.715948 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.715858 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6083807b-72c8-4f79-b464-c17138fd66b2" containerName="gather" Jul 16 18:01:26.715948 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.715933 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6083807b-72c8-4f79-b464-c17138fd66b2" containerName="copy" Jul 16 18:01:26.715948 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.715945 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6083807b-72c8-4f79-b464-c17138fd66b2" containerName="gather" Jul 16 18:01:26.720597 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.720577 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.723914 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.723888 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bn77j\"/\"openshift-service-ca.crt\"" Jul 16 18:01:26.724047 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.723923 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bn77j\"/\"kube-root-ca.crt\"" Jul 16 18:01:26.724173 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.724156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-podres\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.724226 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.724197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-lib-modules\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.724270 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.724243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kvx\" (UniqueName: \"kubernetes.io/projected/671c38d7-b908-4576-8dd6-3b3ba45d5f62-kube-api-access-m7kvx\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.724313 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.724276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-proc\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.724349 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.724323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-sys\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.724963 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.724946 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bn77j\"/\"default-dockercfg-hczwl\"" Jul 16 18:01:26.733643 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.733620 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct"] Jul 16 18:01:26.825512 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.825471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-sys\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.825703 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.825536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-podres\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.825703 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.825569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-lib-modules\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.825703 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.825597 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-sys\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.825703 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.825608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kvx\" (UniqueName: \"kubernetes.io/projected/671c38d7-b908-4576-8dd6-3b3ba45d5f62-kube-api-access-m7kvx\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.825703 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.825658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-proc\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.825931 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.825716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-lib-modules\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.825931 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.825735 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-proc\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.825931 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.825719 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/671c38d7-b908-4576-8dd6-3b3ba45d5f62-podres\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.839263 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.839228 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kvx\" (UniqueName: \"kubernetes.io/projected/671c38d7-b908-4576-8dd6-3b3ba45d5f62-kube-api-access-m7kvx\") pod \"perf-node-gather-daemonset-gk4ct\" (UID: \"671c38d7-b908-4576-8dd6-3b3ba45d5f62\") " pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:26.841660 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:26.841645 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-nj7vd_8c5557a6-144f-4be6-8e3f-1bab640da344/download-server/0.log" Jul 16 18:01:27.030446 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:27.030348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:27.166419 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:27.166384 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct"] Jul 16 18:01:27.169624 ip-10-0-131-83 kubenswrapper[2565]: W0716 18:01:27.169592 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod671c38d7_b908_4576_8dd6_3b3ba45d5f62.slice/crio-7d56217d526dda99e17c8663ab12af9b64915190c96b1f4793224f51b496a0ac WatchSource:0}: Error finding container 7d56217d526dda99e17c8663ab12af9b64915190c96b1f4793224f51b496a0ac: Status 404 returned error can't find the container with id 7d56217d526dda99e17c8663ab12af9b64915190c96b1f4793224f51b496a0ac Jul 16 18:01:27.245392 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:27.245356 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" event={"ID":"671c38d7-b908-4576-8dd6-3b3ba45d5f62","Type":"ContainerStarted","Data":"7d56217d526dda99e17c8663ab12af9b64915190c96b1f4793224f51b496a0ac"} Jul 16 18:01:27.379121 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:27.379097 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-gfvgw_be313ab4-6436-466f-84db-20042d0ddd4a/volume-data-source-validator/0.log" Jul 16 18:01:28.250676 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:28.250641 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" event={"ID":"671c38d7-b908-4576-8dd6-3b3ba45d5f62","Type":"ContainerStarted","Data":"16977c72c7b9d5927b207b318c26cdce62049349da42e94a90e83f24535a1b24"} Jul 16 18:01:28.251073 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:28.250756 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:28.270007 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:28.269961 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" podStartSLOduration=2.269949285 podStartE2EDuration="2.269949285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:01:28.268750751 +0000 UTC m=+1868.940220955" watchObservedRunningTime="2026-07-16 18:01:28.269949285 +0000 UTC m=+1868.941419529" Jul 16 18:01:28.303948 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:28.303921 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mspz8_35a9d4d9-4310-448d-be02-499a71b76856/dns/0.log" Jul 16 18:01:28.329999 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:28.329975 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mspz8_35a9d4d9-4310-448d-be02-499a71b76856/kube-rbac-proxy/0.log" Jul 16 18:01:28.441997 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:28.441967 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fbsff_ae18d2a8-cbb4-4632-9b9e-0135a3d51768/dns-node-resolver/0.log" Jul 16 18:01:28.910817 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:28.910788 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cr9ts_5d4dfa86-8906-4d95-9195-734bfcc49c84/node-ca/0.log" Jul 16 18:01:29.728097 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:29.728068 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-748b65cd7d-mqsgk_c21e59b9-8163-4bbc-bfc6-44abb123817f/router/0.log" Jul 16 18:01:30.170056 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:30.170025 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fb9fq_25c9bc80-1e04-45d6-8f5c-fe45c84ce3fa/serve-healthcheck-canary/0.log" Jul 16 18:01:30.525134 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:30.525046 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-mbmr9_86d1a4e2-abcd-4c3f-947d-1cd87cca019d/insights-operator/0.log" Jul 16 18:01:30.525308 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:30.525152 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-mbmr9_86d1a4e2-abcd-4c3f-947d-1cd87cca019d/insights-operator/1.log" Jul 16 18:01:30.712248 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:30.712220 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7pfv_eb032e05-63e3-44ec-b314-e7839c25ad0f/kube-rbac-proxy/0.log" Jul 16 18:01:30.737539 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:30.737515 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7pfv_eb032e05-63e3-44ec-b314-e7839c25ad0f/exporter/0.log" Jul 16 18:01:30.763412 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:30.763391 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7pfv_eb032e05-63e3-44ec-b314-e7839c25ad0f/extractor/0.log" Jul 16 18:01:34.265472 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:34.265447 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bn77j/perf-node-gather-daemonset-gk4ct" Jul 16 18:01:35.662073 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:35.661990 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-mxfhw_673405af-1260-4611-b0c9-f7ace3907095/kube-storage-version-migrator-operator/1.log" Jul 16 18:01:35.662679 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:35.662658 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-mxfhw_673405af-1260-4611-b0c9-f7ace3907095/kube-storage-version-migrator-operator/0.log" Jul 16 18:01:36.707481 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:36.707453 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c94x7_b46a4bae-3afa-45d4-8f34-b30e97e381fa/kube-multus-additional-cni-plugins/0.log" Jul 16 18:01:36.738388 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:36.738368 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c94x7_b46a4bae-3afa-45d4-8f34-b30e97e381fa/egress-router-binary-copy/0.log" Jul 16 18:01:36.766975 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:36.766897 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c94x7_b46a4bae-3afa-45d4-8f34-b30e97e381fa/cni-plugins/0.log" Jul 16 18:01:36.796337 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:36.796321 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c94x7_b46a4bae-3afa-45d4-8f34-b30e97e381fa/bond-cni-plugin/0.log" Jul 16 18:01:36.825060 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:36.825036 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c94x7_b46a4bae-3afa-45d4-8f34-b30e97e381fa/routeoverride-cni/0.log" Jul 16 18:01:36.852646 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:36.852621 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c94x7_b46a4bae-3afa-45d4-8f34-b30e97e381fa/whereabouts-cni-bincopy/0.log" Jul 16 18:01:36.877311 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:36.877289 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c94x7_b46a4bae-3afa-45d4-8f34-b30e97e381fa/whereabouts-cni/0.log" Jul 16 18:01:37.315935 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:37.315907 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hw446_c4a102ac-4212-4e83-a108-e2992a5007ff/kube-multus/0.log" Jul 16 18:01:37.470148 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:37.470124 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wtj6z_0d1fc925-7c4d-4e7a-9bb1-8607d8524be7/network-metrics-daemon/0.log" Jul 16 18:01:37.495900 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:37.495870 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wtj6z_0d1fc925-7c4d-4e7a-9bb1-8607d8524be7/kube-rbac-proxy/0.log" Jul 16 18:01:38.667598 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:38.667562 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk57w_50e0d455-a823-47e2-881b-bc2fadf9857b/ovn-controller/0.log" Jul 16 18:01:38.689349 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:38.689325 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk57w_50e0d455-a823-47e2-881b-bc2fadf9857b/ovn-acl-logging/0.log" Jul 16 18:01:38.715152 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:38.715126 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk57w_50e0d455-a823-47e2-881b-bc2fadf9857b/kube-rbac-proxy-node/0.log" Jul 16 18:01:38.745875 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:38.745823 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk57w_50e0d455-a823-47e2-881b-bc2fadf9857b/kube-rbac-proxy-ovn-metrics/0.log" Jul 16 18:01:38.772242 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:38.772217 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk57w_50e0d455-a823-47e2-881b-bc2fadf9857b/northd/0.log" Jul 16 18:01:38.797015 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:38.796991 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk57w_50e0d455-a823-47e2-881b-bc2fadf9857b/nbdb/0.log" Jul 16 18:01:38.823883 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:38.823862 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk57w_50e0d455-a823-47e2-881b-bc2fadf9857b/sbdb/0.log" Jul 16 18:01:38.874104 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:38.874074 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk57w_50e0d455-a823-47e2-881b-bc2fadf9857b/ovnkube-controller/0.log" Jul 16 18:01:40.126984 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:40.126937 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-869dcfcdcd-5zxhn_50244332-3ed3-4a66-93a7-e78a83546fe3/check-endpoints/0.log" Jul 16 18:01:40.155954 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:40.155928 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9mcrs_86638b4f-8f78-4423-ac50-0c3216994241/network-check-target-container/0.log" Jul 16 18:01:41.098097 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:41.098032 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4xqgw_4a86433b-1109-40d7-914e-4ef577d6b404/iptables-alerter/0.log" Jul 16 18:01:41.746428 ip-10-0-131-83 kubenswrapper[2565]: I0716 18:01:41.746400 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-7xt6c_4fd3a957-4163-4eaa-93f3-1c81207b8ffa/tuned/0.log"