Jul 08 01:40:42.379088 ip-10-0-134-84 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 08 01:40:42.379216 ip-10-0-134-84 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 08 01:40:42.379308 ip-10-0-134-84 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 08 01:40:42.379603 ip-10-0-134-84 systemd[1]: Failed to start Kubernetes Kubelet. Jul 08 01:40:52.482039 ip-10-0-134-84 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 08 01:40:52.482059 ip-10-0-134-84 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9d96babfeb474b018ca083fb8ca0f0f7 -- Jul 08 01:42:49.218744 ip-10-0-134-84 systemd[1]: Starting Kubernetes Kubelet... Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.613827 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618463 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618490 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618494 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618497 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618500 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618503 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618506 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 01:42:49.699833 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618511 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 01:42:49.693002 ip-10-0-134-84 systemd[1]: Started Kubernetes Kubelet. Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618513 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618516 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618518 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618521 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618524 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618527 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618529 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618532 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618535 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618538 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618541 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618544 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618547 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618550 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618553 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618560 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618563 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618565 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618568 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 01:42:49.700948 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618571 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618574 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618576 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618579 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618582 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618584 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618587 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618590 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618592 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618595 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618597 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618600 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618603 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618605 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618608 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618610 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618614 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618616 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618619 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618621 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 01:42:49.702298 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618624 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618627 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618629 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618632 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618634 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618637 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618639 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618642 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618645 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618648 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618650 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618653 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618656 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618659 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618661 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618663 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618666 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618668 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618671 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618673 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 01:42:49.702942 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618676 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618679 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618681 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618685 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618689 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618693 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618698 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618701 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618706 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618710 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618713 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618717 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618719 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618722 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618725 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618727 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618730 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618732 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 01:42:49.703582 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618735 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618737 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618739 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618742 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618745 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618748 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618751 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618754 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618756 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618759 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618762 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618765 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618768 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.618770 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619521 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619529 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619532 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619535 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619538 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619541 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 01:42:49.704184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619543 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619546 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619548 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619552 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619555 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619557 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619561 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619564 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619567 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619569 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619572 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619575 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619577 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619580 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619582 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619585 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619587 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619591 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619594 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619597 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 01:42:49.704980 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619599 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619602 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619604 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619607 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619609 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619612 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619614 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619617 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619619 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619622 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619624 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619627 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619629 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619632 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619635 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619637 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619640 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619643 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619646 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619649 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 01:42:49.705955 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619652 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619655 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619657 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619660 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619663 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619665 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619668 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619670 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619673 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619676 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619679 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619682 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619684 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619687 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619689 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619692 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619694 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619696 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619699 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 01:42:49.706925 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619703 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619707 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619713 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619716 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619719 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619722 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619725 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619728 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619731 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619734 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619738 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619740 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619743 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619745 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619748 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619751 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619754 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619758 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619761 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 01:42:49.707877 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619764 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619767 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619771 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619773 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619776 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619778 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619784 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619786 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619789 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619792 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619796 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619799 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619801 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619804 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.619806 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620185 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620189 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620192 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620194 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620197 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 01:42:49.708819 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620199 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620202 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620205 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620208 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620210 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620213 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620215 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620218 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620220 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620223 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620225 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620228 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620230 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620233 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620235 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620238 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620240 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620243 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620248 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620250 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 01:42:49.869064 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620253 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620255 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620259 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620262 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620265 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620267 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620270 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620272 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620275 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620277 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620280 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620282 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620284 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620287 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620289 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620292 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620295 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620298 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620300 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 01:42:49.884121 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620304 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620306 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620308 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620311 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620313 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620316 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620318 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620320 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620323 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620326 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620328 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620332 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620335 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620337 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620340 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620344 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620346 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620349 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620351 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620354 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 01:42:49.885073 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620356 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620359 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620361 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620364 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620367 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620369 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620372 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620375 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620379 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620381 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620384 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620386 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620389 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620391 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620394 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620396 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620399 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620401 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620404 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620406 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 01:42:49.885917 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620410 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620412 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620415 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620419 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620422 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620424 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620427 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620432 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620436 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620440 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620443 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620445 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620448 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620450 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.620453 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620553 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620564 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620573 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620579 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620585 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 01:42:49.886810 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620590 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620596 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620601 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620605 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620608 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620611 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620614 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620618 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620620 2578 flags.go:64] FLAG: --cgroup-root="" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620623 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620626 2578 flags.go:64] FLAG: --client-ca-file="" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620629 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620632 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620637 2578 flags.go:64] FLAG: --cluster-domain="" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620640 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620644 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620647 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620651 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620654 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620662 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620665 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620668 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620671 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620674 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 01:42:49.887479 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620677 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620681 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620684 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620688 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620691 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620694 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620697 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620699 2578 flags.go:64] FLAG: --enable-server="true" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620702 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620707 2578 flags.go:64] FLAG: --event-burst="100" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620710 2578 flags.go:64] FLAG: --event-qps="50" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620713 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620717 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620720 2578 flags.go:64] FLAG: --eviction-hard="" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620723 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620726 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620729 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620732 2578 flags.go:64] FLAG: --eviction-soft="" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620735 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620738 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620741 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620743 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620746 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620750 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620753 2578 flags.go:64] FLAG: --feature-gates="" Jul 08 01:42:49.888376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620757 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620760 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620765 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620768 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620771 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620774 2578 flags.go:64] FLAG: --help="false" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620777 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-84.ec2.internal" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620781 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620784 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620786 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620790 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620793 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620796 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620799 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620802 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620805 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620808 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620810 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620813 2578 flags.go:64] FLAG: --kube-reserved="" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620816 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620819 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620822 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620825 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620828 2578 flags.go:64] FLAG: --lock-file="" Jul 08 01:42:49.889283 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620831 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620833 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620836 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620842 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620844 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620847 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620852 2578 flags.go:64] FLAG: --logging-format="text" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620855 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620858 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620861 2578 flags.go:64] FLAG: --manifest-url="" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620865 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620870 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620872 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620877 2578 flags.go:64] FLAG: --max-pods="110" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620880 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620883 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620885 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620888 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620891 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620894 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620897 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620904 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620907 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620910 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 01:42:49.890448 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620913 2578 flags.go:64] FLAG: --pod-cidr="" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620916 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620922 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620925 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620928 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620932 2578 flags.go:64] FLAG: --port="10250" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620935 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620938 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08f1b8abcd38305bc" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620941 2578 flags.go:64] FLAG: --qos-reserved="" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620944 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620947 2578 flags.go:64] FLAG: --register-node="true" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620950 2578 flags.go:64] FLAG: --register-with-taints="" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620954 2578 flags.go:64] FLAG: --registry-burst="10" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620957 2578 flags.go:64] FLAG: --registry-qps="5" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620960 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620964 2578 flags.go:64] FLAG: --reserved-memory="" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620968 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620971 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620977 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620980 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620983 2578 flags.go:64] FLAG: --runonce="false" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620985 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620989 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620992 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620995 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 01:42:49.891325 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.620998 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621001 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621004 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621007 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621009 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621013 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621015 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621018 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621021 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621024 2578 flags.go:64] FLAG: --system-cgroups="" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621027 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621032 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621035 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621038 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621043 2578 flags.go:64] FLAG: --tls-min-version="" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621046 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621048 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621051 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621054 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621057 2578 flags.go:64] FLAG: --v="2" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621061 2578 flags.go:64] FLAG: --version="false" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621064 2578 flags.go:64] FLAG: --vmodule="" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621070 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.621073 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621191 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 01:42:49.892229 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621195 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621199 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621203 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621205 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621208 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621211 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621214 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621217 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621220 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621222 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621225 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621228 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621231 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621233 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621236 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621239 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621241 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621244 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621247 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621249 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 01:42:49.893326 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621252 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621255 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621258 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621260 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621263 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621266 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621268 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621271 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621274 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621278 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621280 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621283 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621287 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621290 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621292 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621295 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621297 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621300 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621303 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621305 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 01:42:49.894063 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621308 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621310 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621313 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621315 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621318 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621320 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621323 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621326 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621328 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621331 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621333 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621336 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621338 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621341 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621344 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621347 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621349 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621352 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621356 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 01:42:50.026007 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621359 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621362 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621366 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621369 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621372 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621376 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621379 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621381 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621384 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621387 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621389 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621391 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621395 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621399 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621401 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621404 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621406 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621409 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621411 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621414 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 01:42:50.027313 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621416 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621419 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621421 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621424 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621426 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621429 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621431 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621434 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621437 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621439 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621442 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621444 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621447 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621449 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621454 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621457 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621461 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621465 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 01:42:50.027940 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.621468 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.622242 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.629679 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.629696 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.629716 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.629725 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629782 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629787 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629792 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629795 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629798 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629801 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629803 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629807 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629811 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 01:42:50.028522 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629815 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629818 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629821 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629824 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629827 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629829 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629832 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629835 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629837 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629840 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629842 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629845 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629848 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629850 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629854 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629857 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629859 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629862 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629865 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629868 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 01:42:50.029011 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629871 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629874 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629876 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629881 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629884 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629886 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629889 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629892 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629894 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629897 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629901 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629904 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629907 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629910 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629913 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629916 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629919 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629922 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629924 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629927 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 01:42:50.029720 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629929 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629932 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629934 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629937 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629939 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629942 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629944 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629947 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629950 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629955 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629957 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629960 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629962 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629966 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629968 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629973 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629976 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629978 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629980 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629983 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 01:42:50.030295 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629985 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629988 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629990 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629993 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629995 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.629998 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630001 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630003 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630006 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630008 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630011 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630014 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630016 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630019 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630021 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630024 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630026 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630029 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630031 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 01:42:50.030928 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630034 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630036 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630040 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630043 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630046 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630050 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630054 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630057 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630061 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630064 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630066 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.630071 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630184 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630189 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630191 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 01:42:50.031571 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630194 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630198 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630202 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630205 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630208 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630211 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630214 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630217 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630219 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630222 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630225 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630227 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630230 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630232 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630235 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630238 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630240 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630242 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630245 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 01:42:50.032068 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630247 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630251 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630254 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630256 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630259 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630262 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630266 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630269 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630272 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630274 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630277 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630279 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630282 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630284 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630287 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630289 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630292 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630294 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630297 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630299 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 01:42:50.032646 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630302 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630305 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630308 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630310 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630313 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630316 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630319 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630322 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630331 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630335 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630338 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630341 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630343 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630347 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630349 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630352 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630355 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630358 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630361 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630365 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 01:42:50.033287 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630368 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630370 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630373 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630375 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630379 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630383 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630386 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630388 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630390 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630393 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630396 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630398 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630401 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630403 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630406 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630409 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630411 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630414 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630417 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 01:42:50.033898 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630419 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630422 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630424 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630427 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630430 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630432 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630438 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630441 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630444 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630446 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630448 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630453 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630455 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630458 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630461 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630463 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630466 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 01:42:50.034485 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:49.630469 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.630488 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.630614 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.634023 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.635574 2578 server.go:1019] "Starting client certificate rotation" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.638424 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.638859 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.638932 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.644451 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.652671 2578 log.go:25] "Validated CRI v1 runtime API" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.655073 2578 log.go:25] "Validated CRI v1 image API" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.656581 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 01:42:50.034991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.660169 2578 fs.go:135] Filesystem UUIDs: map[34450b77-b3eb-48b5-9cc3-9b936b5459cd:/dev/nvme0n1p3 4919e7f4-9fc7-4f6f-937e-19d0d1b4d7ff:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 08 01:42:50.035417 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.660185 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 01:42:50.035494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.665037 2578 manager.go:216] Machine: {Timestamp:2026-07-08 01:42:49.66375195 +0000 UTC m=+0.325666478 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198308 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b37c3a4639f536df40161d71eeb36 SystemUUID:ec2b37c3-a463-9f53-6df4-0161d71eeb36 BootID:9d96babf-eb47-4b01-8ca0-83fb8ca0f0f7 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:40:66:82:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:40:66:82:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:7f:69:d9:30:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 01:42:50.035494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.665141 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 01:42:50.035494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.665256 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 01:42:50.035494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.668570 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 01:42:50.035494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.668594 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 01:42:50.035494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.670107 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 01:42:50.035494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.670118 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 01:42:50.035494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.670126 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.670939 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.672688 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.676550 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.676661 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.676688 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.676697 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.676715 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.676724 2578 kubelet.go:398] "Adding apiserver pod source" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.676732 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.677063 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.677078 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.677519 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.683433 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.685928 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.685956 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.686993 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687006 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687013 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687019 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687027 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687032 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687039 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687047 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687057 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687063 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687071 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687077 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687752 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.687761 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.689448 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.689513 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.692085 2578 server.go:1268] "Started kubelet" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.692193 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.692192 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.692301 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.694195 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.696280 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.697928 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 01:42:50.035853 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.699579 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-84.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.699730 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-84.ec2.internal.18c02d09224b06a1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-84.ec2.internal,UID:ip-10-0-134-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-84.ec2.internal,},FirstTimestamp:2026-07-08 01:42:49.692014241 +0000 UTC m=+0.353928769,LastTimestamp:2026-07-08 01:42:49.692014241 +0000 UTC m=+0.353928769,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-84.ec2.internal,}" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.702067 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6gdvz" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.704377 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.704391 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.704461 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705155 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705152 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705179 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705183 2578 factory.go:55] Registering systemd factory Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705197 2578 factory.go:223] Registration of the systemd container factory successfully Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705253 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705264 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705400 2578 factory.go:153] Registering CRI-O factory Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705415 2578 factory.go:223] Registration of the crio container factory successfully Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705467 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705514 2578 factory.go:103] Registering Raw factory Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705528 2578 manager.go:1196] Started watching for new ooms in manager Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.705955 2578 manager.go:349] Starting recovery of all containers Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.708655 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6gdvz" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.712586 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.720572 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.720681 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.725526 2578 manager.go:354] Recovery completed Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.730213 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.732465 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.732510 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.732524 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.733039 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.733048 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.733050 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.733115 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.733778 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory" Jul 08 01:42:50.036970 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.733803 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.733820 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.734511 2578 policy_none.go:49] "None policy: Start" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.734523 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.734535 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.735737 2578 policy_none.go:47] "Start" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.773895 2578 manager.go:337] "Starting Device Plugin manager" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.773929 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.773940 2578 server.go:87] "Starting device plugin registration server" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.774187 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.774199 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.774295 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.774377 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.774385 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.775001 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.775231 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.876100 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.878125 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.878154 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.878165 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.878191 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.891468 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.891489 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-84.ec2.internal\": node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.899816 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.899850 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.899870 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.899901 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:49.901887 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:49.912074 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.000334 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"] Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.000422 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.002203 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.002231 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.002240 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.003680 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.003787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.003813 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.004580 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory" Jul 08 01:42:50.038132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.004604 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.004613 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.004619 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.004630 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.004643 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.005976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.006005 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.006670 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.006695 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.006706 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.007813 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e64f5c6be32d9511c588a4d2d5ea69a6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-84.ec2.internal\" (UID: \"e64f5c6be32d9511c588a4d2d5ea69a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.007835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cf5b13b5c7134c2ef58eddb3c8605ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"0cf5b13b5c7134c2ef58eddb3c8605ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.007852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cf5b13b5c7134c2ef58eddb3c8605ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"0cf5b13b5c7134c2ef58eddb3c8605ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.012950 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.039603 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.039307 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-84.ec2.internal\" not found" node="ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.233205 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.044347 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-84.ec2.internal\" not found" node="ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.233205 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.108566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e64f5c6be32d9511c588a4d2d5ea69a6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-84.ec2.internal\" (UID: \"e64f5c6be32d9511c588a4d2d5ea69a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.233205 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.108592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cf5b13b5c7134c2ef58eddb3c8605ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"0cf5b13b5c7134c2ef58eddb3c8605ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.233205 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.108618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cf5b13b5c7134c2ef58eddb3c8605ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"0cf5b13b5c7134c2ef58eddb3c8605ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.233205 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.108663 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cf5b13b5c7134c2ef58eddb3c8605ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"0cf5b13b5c7134c2ef58eddb3c8605ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.233205 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.108676 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e64f5c6be32d9511c588a4d2d5ea69a6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-84.ec2.internal\" (UID: \"e64f5c6be32d9511c588a4d2d5ea69a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.233205 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.108680 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cf5b13b5c7134c2ef58eddb3c8605ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"0cf5b13b5c7134c2ef58eddb3c8605ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.233205 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.113122 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.233205 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.214248 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.398515 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.314560 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.398515 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.346086 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.398515 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.348865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.579335 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.414963 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.579335 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.516009 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.742262 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.601794 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 01:42:50.742262 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.616910 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.742262 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.639769 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 01:42:50.742262 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.639903 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 01:42:50.742262 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.639952 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 01:42:50.742262 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:50.717256 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found" Jul 08 01:42:50.742262 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.719575 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 01:37:50 +0000 UTC" deadline="2028-02-11 17:49:13.360510158 +0000 UTC" Jul 08 01:42:50.742262 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.719599 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14008h6m22.640914825s" Jul 08 01:42:50.742262 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.721833 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 01:42:50.899201 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.804625 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.899201 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.814589 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 01:42:50.899201 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.815874 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" Jul 08 01:42:50.899201 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.826956 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 01:42:50.899201 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:50.881772 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 01:42:51.158554 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:51.067890 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64f5c6be32d9511c588a4d2d5ea69a6.slice/crio-9d1ad6a952138806466cb7c61a9871b9259a78a01507bdfbdfe24d45b8205d61 WatchSource:0}: Error finding container 9d1ad6a952138806466cb7c61a9871b9259a78a01507bdfbdfe24d45b8205d61: Status 404 returned error can't find the container with id 9d1ad6a952138806466cb7c61a9871b9259a78a01507bdfbdfe24d45b8205d61 Jul 08 01:42:51.158554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.074858 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 01:42:51.158554 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:51.075877 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf5b13b5c7134c2ef58eddb3c8605ed.slice/crio-ea2306cc5f4d6a31fb05ea5c68fef43ba9b163bf532ca2fa069cd84702d87071 WatchSource:0}: Error finding container ea2306cc5f4d6a31fb05ea5c68fef43ba9b163bf532ca2fa069cd84702d87071: Status 404 returned error can't find the container with id ea2306cc5f4d6a31fb05ea5c68fef43ba9b163bf532ca2fa069cd84702d87071 Jul 08 01:42:51.684887 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.678064 2578 apiserver.go:52] "Watching apiserver" Jul 08 01:42:51.686996 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.685477 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 01:42:51.686996 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.686042 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-fm4cg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p","openshift-dns/node-resolver-66zvg","openshift-image-registry/node-ca-54f2j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal","openshift-multus/multus-additional-cni-plugins-7mn7c","openshift-network-diagnostics/network-check-target-62q7j","openshift-network-operator/iptables-alerter-l6grg","kube-system/konnectivity-agent-l7qll","kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qfsjk","openshift-multus/multus-m5xj7","openshift-multus/network-metrics-daemon-852qz","openshift-ovn-kubernetes/ovnkube-node-5r544"] Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.689258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.689353 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.699088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.702543 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vz4kd\"" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.703411 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.703702 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.704462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.705025 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.707984 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.708922 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.709127 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.709333 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-46m2x\"" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.712318 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.714723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.717639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-sysconfig\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-systemd\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af998a88-47fd-4241-bea5-d3cac7487175-etc-tuned\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a542d4a8-3f11-4d97-b235-71de248f8a72-serviceca\") pod \"node-ca-54f2j\" (UID: \"a542d4a8-3f11-4d97-b235-71de248f8a72\") " pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-lib-modules\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-kubernetes\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.758366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-sysctl-d\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-sys\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqb64\" (UniqueName: \"kubernetes.io/projected/af998a88-47fd-4241-bea5-d3cac7487175-kube-api-access-vqb64\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a542d4a8-3f11-4d97-b235-71de248f8a72-host\") pod \"node-ca-54f2j\" (UID: \"a542d4a8-3f11-4d97-b235-71de248f8a72\") " pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-host\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af998a88-47fd-4241-bea5-d3cac7487175-tmp\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/392a0884-e501-4432-870e-bd698aaa7616-hosts-file\") pod \"node-resolver-66zvg\" (UID: \"392a0884-e501-4432-870e-bd698aaa7616\") " pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392a0884-e501-4432-870e-bd698aaa7616-tmp-dir\") pod \"node-resolver-66zvg\" (UID: \"392a0884-e501-4432-870e-bd698aaa7616\") " pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.726915 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.726940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.727766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.727765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2jv\" (UniqueName: \"kubernetes.io/projected/392a0884-e501-4432-870e-bd698aaa7616-kube-api-access-6g2jv\") pod \"node-resolver-66zvg\" (UID: \"392a0884-e501-4432-870e-bd698aaa7616\") " pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.727899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-var-lib-kubelet\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.761220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.727955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-modprobe-d\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.728008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-sysctl-conf\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.728041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-run\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.728075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4rjh\" (UniqueName: \"kubernetes.io/projected/a542d4a8-3f11-4d97-b235-71de248f8a72-kube-api-access-w4rjh\") pod \"node-ca-54f2j\" (UID: \"a542d4a8-3f11-4d97-b235-71de248f8a72\") " pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.728706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.728810 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.731759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.748461 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.748837 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749022 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749119 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kt24m\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749215 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749307 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749387 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749489 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jqkt6\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749596 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749675 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749778 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.749834 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.750134 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.751235 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.752625 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.752821 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.753011 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.753159 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.753475 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.753647 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 01:42:51.763238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.754085 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nxlwx\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.754277 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.754447 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.754490 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.754638 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-twchd\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.754664 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tn5ph\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.754813 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qdd88\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.754981 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.755031 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.755159 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2vx2f\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.755184 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.757867 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 01:42:51.765006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.758042 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cn6hr" Jul 08 01:42:51.806773 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.806747 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 01:42:51.828491 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-cni-netd\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.828618 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828508 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a5d6c6dc-5229-44fb-8744-715743709f96-agent-certs\") pod \"konnectivity-agent-l7qll\" (UID: \"a5d6c6dc-5229-44fb-8744-715743709f96\") " pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:51.828618 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.828757 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/392a0884-e501-4432-870e-bd698aaa7616-hosts-file\") pod \"node-resolver-66zvg\" (UID: \"392a0884-e501-4432-870e-bd698aaa7616\") " pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.828757 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392a0884-e501-4432-870e-bd698aaa7616-tmp-dir\") pod \"node-resolver-66zvg\" (UID: \"392a0884-e501-4432-870e-bd698aaa7616\") " pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.828757 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-var-lib-openvswitch\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.828920 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3fd153b-fb91-454a-9517-828c780a194b-ovn-node-metrics-cert\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.828920 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828794 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/392a0884-e501-4432-870e-bd698aaa7616-hosts-file\") pod \"node-resolver-66zvg\" (UID: \"392a0884-e501-4432-870e-bd698aaa7616\") " pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.828920 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.828920 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3fd153b-fb91-454a-9517-828c780a194b-env-overrides\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.828920 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-system-cni-dir\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.829169 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-cnibin\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.829169 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.828969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-os-release\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.829169 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/529292c4-1ccf-4ebc-a014-af5094659f0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.829169 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a5d6c6dc-5229-44fb-8744-715743709f96-konnectivity-ca\") pod \"konnectivity-agent-l7qll\" (UID: \"a5d6c6dc-5229-44fb-8744-715743709f96\") " pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:51.829169 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392a0884-e501-4432-870e-bd698aaa7616-tmp-dir\") pod \"node-resolver-66zvg\" (UID: \"392a0884-e501-4432-870e-bd698aaa7616\") " pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.829424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-kubelet-config\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:51.829424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-dbus\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:51.829424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-etc-openvswitch\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.829424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-etc-kubernetes\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.829424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.829424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lzq6\" (UniqueName: \"kubernetes.io/projected/48f036fb-7c1c-427c-a1df-0da4575540d0-kube-api-access-2lzq6\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.829424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/529292c4-1ccf-4ebc-a014-af5094659f0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.829424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2jv\" (UniqueName: \"kubernetes.io/projected/392a0884-e501-4432-870e-bd698aaa7616-kube-api-access-6g2jv\") pod \"node-resolver-66zvg\" (UID: \"392a0884-e501-4432-870e-bd698aaa7616\") " pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.829424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829405 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-systemd-units\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.829893 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-os-release\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.830047 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.829974 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-run-netns\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.830047 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-run-multus-certs\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.830047 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-socket-dir-parent\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-run\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4rjh\" (UniqueName: \"kubernetes.io/projected/a542d4a8-3f11-4d97-b235-71de248f8a72-kube-api-access-w4rjh\") pod \"node-ca-54f2j\" (UID: \"a542d4a8-3f11-4d97-b235-71de248f8a72\") " pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3fd153b-fb91-454a-9517-828c780a194b-ovnkube-config\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59ct\" (UniqueName: \"kubernetes.io/projected/85d121a8-c00e-4baf-a410-40d9ac037d1c-kube-api-access-t59ct\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05594ec0-102a-4511-ad57-fb9d94abac4d-iptables-alerter-script\") pod \"iptables-alerter-l6grg\" (UID: \"05594ec0-102a-4511-ad57-fb9d94abac4d\") " pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-cnibin\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-run-systemd\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-run-openvswitch\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830232 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-run\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.830284 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-daemon-config\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-lib-modules\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-sysctl-d\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-sys\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqb64\" (UniqueName: \"kubernetes.io/projected/af998a88-47fd-4241-bea5-d3cac7487175-kube-api-access-vqb64\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-lib-modules\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-sys\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a542d4a8-3f11-4d97-b235-71de248f8a72-host\") pod \"node-ca-54f2j\" (UID: \"a542d4a8-3f11-4d97-b235-71de248f8a72\") " pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-var-lib-cni-bin\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830769 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-sysctl-d\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.830774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-var-lib-cni-multus\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830816 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a542d4a8-3f11-4d97-b235-71de248f8a72-host\") pod \"node-ca-54f2j\" (UID: \"a542d4a8-3f11-4d97-b235-71de248f8a72\") " pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-host\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-slash\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-hostroot\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-kubelet\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-cni-bin\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.830992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-host\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-system-cni-dir\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-cni-dir\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831057 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/061de9e3-7d37-4c96-aece-75722b0c41c8-cni-binary-copy\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-conf-dir\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/529292c4-1ccf-4ebc-a014-af5094659f0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.831370 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-sysconfig\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-systemd\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5w9b\" (UniqueName: \"kubernetes.io/projected/d3fd153b-fb91-454a-9517-828c780a194b-kube-api-access-z5w9b\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf98n\" (UniqueName: \"kubernetes.io/projected/05594ec0-102a-4511-ad57-fb9d94abac4d-kube-api-access-bf98n\") pod \"iptables-alerter-l6grg\" (UID: \"05594ec0-102a-4511-ad57-fb9d94abac4d\") " pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-systemd\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-run-k8s-cni-cncf-io\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-sysconfig\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-socket-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-kubernetes\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-sys-fs\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af998a88-47fd-4241-bea5-d3cac7487175-tmp\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3fd153b-fb91-454a-9517-828c780a194b-ovnkube-script-lib\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.832004 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-device-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-var-lib-kubelet\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-modprobe-d\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-sysctl-conf\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-run-ovn\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-log-socket\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-registration-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831791 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-modprobe-d\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af998a88-47fd-4241-bea5-d3cac7487175-etc-tuned\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a542d4a8-3f11-4d97-b235-71de248f8a72-serviceca\") pod \"node-ca-54f2j\" (UID: \"a542d4a8-3f11-4d97-b235-71de248f8a72\") " pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-node-log\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05594ec0-102a-4511-ad57-fb9d94abac4d-host-slash\") pod \"iptables-alerter-l6grg\" (UID: \"05594ec0-102a-4511-ad57-fb9d94abac4d\") " pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831871 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 01:42:51.832711 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-var-lib-kubelet\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.833431 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slq4l\" (UniqueName: \"kubernetes.io/projected/061de9e3-7d37-4c96-aece-75722b0c41c8-kube-api-access-slq4l\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.833431 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc48k\" (UniqueName: \"kubernetes.io/projected/529292c4-1ccf-4ebc-a014-af5094659f0c-kube-api-access-kc48k\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.833431 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.832022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-run-netns\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.833431 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.831672 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-kubernetes\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.833431 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.832138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-var-lib-kubelet\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.833431 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.832263 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af998a88-47fd-4241-bea5-d3cac7487175-etc-sysctl-conf\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.833431 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.832569 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a542d4a8-3f11-4d97-b235-71de248f8a72-serviceca\") pod \"node-ca-54f2j\" (UID: \"a542d4a8-3f11-4d97-b235-71de248f8a72\") " pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.837173 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.836735 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 01:42:51.839244 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.839223 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 01:42:51.840135 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.839252 2578 projected.go:196] Error preparing data for projected volume kube-api-access-ttww6 for pod openshift-network-diagnostics/network-check-target-62q7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:51.840237 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.840222 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6 podName:24490e15-c228-4567-a883-c504c7cb6975 nodeName:}" failed. No retries permitted until 2026-07-08 01:42:52.340196585 +0000 UTC m=+3.002111108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ttww6" (UniqueName: "kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6") pod "network-check-target-62q7j" (UID: "24490e15-c228-4567-a883-c504c7cb6975") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:51.840237 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.839642 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af998a88-47fd-4241-bea5-d3cac7487175-etc-tuned\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.840367 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.840087 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4rjh\" (UniqueName: \"kubernetes.io/projected/a542d4a8-3f11-4d97-b235-71de248f8a72-kube-api-access-w4rjh\") pod \"node-ca-54f2j\" (UID: \"a542d4a8-3f11-4d97-b235-71de248f8a72\") " pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:51.840367 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.838049 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af998a88-47fd-4241-bea5-d3cac7487175-tmp\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.840465 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.840436 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqb64\" (UniqueName: \"kubernetes.io/projected/af998a88-47fd-4241-bea5-d3cac7487175-kube-api-access-vqb64\") pod \"tuned-qfsjk\" (UID: \"af998a88-47fd-4241-bea5-d3cac7487175\") " pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:51.841634 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.841609 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2jv\" (UniqueName: \"kubernetes.io/projected/392a0884-e501-4432-870e-bd698aaa7616-kube-api-access-6g2jv\") pod \"node-resolver-66zvg\" (UID: \"392a0884-e501-4432-870e-bd698aaa7616\") " pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:51.907244 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.907190 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" event={"ID":"0cf5b13b5c7134c2ef58eddb3c8605ed","Type":"ContainerStarted","Data":"ea2306cc5f4d6a31fb05ea5c68fef43ba9b163bf532ca2fa069cd84702d87071"} Jul 08 01:42:51.907695 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.907668 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" event={"ID":"e64f5c6be32d9511c588a4d2d5ea69a6","Type":"ContainerStarted","Data":"9d1ad6a952138806466cb7c61a9871b9259a78a01507bdfbdfe24d45b8205d61"} Jul 08 01:42:51.932415 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-var-lib-cni-bin\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932581 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-var-lib-cni-multus\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932581 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-slash\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.932581 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-var-lib-cni-bin\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932581 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-hostroot\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-kubelet\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-cni-bin\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932624 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-var-lib-cni-multus\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-hostroot\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-slash\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932673 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-kubelet\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-system-cni-dir\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-cni-bin\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932734 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-system-cni-dir\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-cni-dir\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/061de9e3-7d37-4c96-aece-75722b0c41c8-cni-binary-copy\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.932804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-conf-dir\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/529292c4-1ccf-4ebc-a014-af5094659f0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5w9b\" (UniqueName: \"kubernetes.io/projected/d3fd153b-fb91-454a-9517-828c780a194b-kube-api-access-z5w9b\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932916 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-cni-dir\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf98n\" (UniqueName: \"kubernetes.io/projected/05594ec0-102a-4511-ad57-fb9d94abac4d-kube-api-access-bf98n\") pod \"iptables-alerter-l6grg\" (UID: \"05594ec0-102a-4511-ad57-fb9d94abac4d\") " pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-run-k8s-cni-cncf-io\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.933012 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.932995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-socket-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-sys-fs\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.933075 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs podName:85d121a8-c00e-4baf-a410-40d9ac037d1c nodeName:}" failed. No retries permitted until 2026-07-08 01:42:52.433054146 +0000 UTC m=+3.094968682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs") pod "network-metrics-daemon-852qz" (UID: "85d121a8-c00e-4baf-a410-40d9ac037d1c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933102 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-run-k8s-cni-cncf-io\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.933597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3fd153b-fb91-454a-9517-828c780a194b-ovnkube-script-lib\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-device-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933217 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-socket-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-run-ovn\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-log-socket\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-registration-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-node-log\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05594ec0-102a-4511-ad57-fb9d94abac4d-host-slash\") pod \"iptables-alerter-l6grg\" (UID: \"05594ec0-102a-4511-ad57-fb9d94abac4d\") " pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/061de9e3-7d37-4c96-aece-75722b0c41c8-cni-binary-copy\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933105 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-sys-fs\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-var-lib-kubelet\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slq4l\" (UniqueName: \"kubernetes.io/projected/061de9e3-7d37-4c96-aece-75722b0c41c8-kube-api-access-slq4l\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.933477 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc48k\" (UniqueName: \"kubernetes.io/projected/529292c4-1ccf-4ebc-a014-af5094659f0c-kube-api-access-kc48k\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.934381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-run-netns\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-log-socket\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933273 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-conf-dir\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:51.933562 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret podName:1a6bb622-0bb3-4f7a-85bd-5714b6756d82 nodeName:}" failed. No retries permitted until 2026-07-08 01:42:52.433529625 +0000 UTC m=+3.095444157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret") pod "global-pull-secret-syncer-fm4cg" (UID: "1a6bb622-0bb3-4f7a-85bd-5714b6756d82") : object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933579 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-run-netns\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-cni-netd\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933603 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-node-log\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933605 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05594ec0-102a-4511-ad57-fb9d94abac4d-host-slash\") pod \"iptables-alerter-l6grg\" (UID: \"05594ec0-102a-4511-ad57-fb9d94abac4d\") " pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a5d6c6dc-5229-44fb-8744-715743709f96-agent-certs\") pod \"konnectivity-agent-l7qll\" (UID: \"a5d6c6dc-5229-44fb-8744-715743709f96\") " pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933632 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-host-cni-netd\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-var-lib-kubelet\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933671 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-registration-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933677 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-run-ovn\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935211 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-var-lib-openvswitch\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3fd153b-fb91-454a-9517-828c780a194b-ovn-node-metrics-cert\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3fd153b-fb91-454a-9517-828c780a194b-ovnkube-script-lib\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933723 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-device-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3fd153b-fb91-454a-9517-828c780a194b-env-overrides\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-system-cni-dir\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-cnibin\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-os-release\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/529292c4-1ccf-4ebc-a014-af5094659f0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a5d6c6dc-5229-44fb-8744-715743709f96-konnectivity-ca\") pod \"konnectivity-agent-l7qll\" (UID: \"a5d6c6dc-5229-44fb-8744-715743709f96\") " pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933979 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-cnibin\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-kubelet-config\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:51.935918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933713 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/529292c4-1ccf-4ebc-a014-af5094659f0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-dbus\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934025 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-system-cni-dir\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.933790 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-var-lib-openvswitch\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-etc-openvswitch\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-etc-kubernetes\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934112 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-kubelet-config\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934114 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-os-release\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-etc-kubernetes\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-dbus\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48f036fb-7c1c-427c-a1df-0da4575540d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934191 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3fd153b-fb91-454a-9517-828c780a194b-env-overrides\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-etc-openvswitch\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.936621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lzq6\" (UniqueName: \"kubernetes.io/projected/48f036fb-7c1c-427c-a1df-0da4575540d0-kube-api-access-2lzq6\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/529292c4-1ccf-4ebc-a014-af5094659f0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-systemd-units\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-os-release\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-run-netns\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-run-multus-certs\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-socket-dir-parent\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3fd153b-fb91-454a-9517-828c780a194b-ovnkube-config\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934414 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/529292c4-1ccf-4ebc-a014-af5094659f0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t59ct\" (UniqueName: \"kubernetes.io/projected/85d121a8-c00e-4baf-a410-40d9ac037d1c-kube-api-access-t59ct\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05594ec0-102a-4511-ad57-fb9d94abac4d-iptables-alerter-script\") pod \"iptables-alerter-l6grg\" (UID: \"05594ec0-102a-4511-ad57-fb9d94abac4d\") " pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-cnibin\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934507 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-os-release\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-run-systemd\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-run-openvswitch\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.937292 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-daemon-config\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/529292c4-1ccf-4ebc-a014-af5094659f0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934839 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-run-netns\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-host-run-multus-certs\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.934941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-socket-dir-parent\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.935155 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-systemd-units\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.935162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/061de9e3-7d37-4c96-aece-75722b0c41c8-multus-daemon-config\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.935232 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a5d6c6dc-5229-44fb-8744-715743709f96-konnectivity-ca\") pod \"konnectivity-agent-l7qll\" (UID: \"a5d6c6dc-5229-44fb-8744-715743709f96\") " pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.935247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/061de9e3-7d37-4c96-aece-75722b0c41c8-cnibin\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.935253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/529292c4-1ccf-4ebc-a014-af5094659f0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.935307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-run-systemd\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.935316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3fd153b-fb91-454a-9517-828c780a194b-run-openvswitch\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.935430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3fd153b-fb91-454a-9517-828c780a194b-ovnkube-config\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.935675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05594ec0-102a-4511-ad57-fb9d94abac4d-iptables-alerter-script\") pod \"iptables-alerter-l6grg\" (UID: \"05594ec0-102a-4511-ad57-fb9d94abac4d\") " pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.936899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a5d6c6dc-5229-44fb-8744-715743709f96-agent-certs\") pod \"konnectivity-agent-l7qll\" (UID: \"a5d6c6dc-5229-44fb-8744-715743709f96\") " pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:51.937864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.937138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3fd153b-fb91-454a-9517-828c780a194b-ovn-node-metrics-cert\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:51.943712 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.943670 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc48k\" (UniqueName: \"kubernetes.io/projected/529292c4-1ccf-4ebc-a014-af5094659f0c-kube-api-access-kc48k\") pod \"multus-additional-cni-plugins-7mn7c\" (UID: \"529292c4-1ccf-4ebc-a014-af5094659f0c\") " pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:51.945215 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.945190 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf98n\" (UniqueName: \"kubernetes.io/projected/05594ec0-102a-4511-ad57-fb9d94abac4d-kube-api-access-bf98n\") pod \"iptables-alerter-l6grg\" (UID: \"05594ec0-102a-4511-ad57-fb9d94abac4d\") " pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:51.945537 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.945509 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59ct\" (UniqueName: \"kubernetes.io/projected/85d121a8-c00e-4baf-a410-40d9ac037d1c-kube-api-access-t59ct\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:51.946524 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.946493 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slq4l\" (UniqueName: \"kubernetes.io/projected/061de9e3-7d37-4c96-aece-75722b0c41c8-kube-api-access-slq4l\") pod \"multus-m5xj7\" (UID: \"061de9e3-7d37-4c96-aece-75722b0c41c8\") " pod="openshift-multus/multus-m5xj7" Jul 08 01:42:51.947419 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.947392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lzq6\" (UniqueName: \"kubernetes.io/projected/48f036fb-7c1c-427c-a1df-0da4575540d0-kube-api-access-2lzq6\") pod \"aws-ebs-csi-driver-node-h969p\" (UID: \"48f036fb-7c1c-427c-a1df-0da4575540d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:51.948056 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:51.947988 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5w9b\" (UniqueName: \"kubernetes.io/projected/d3fd153b-fb91-454a-9517-828c780a194b-kube-api-access-z5w9b\") pod \"ovnkube-node-5r544\" (UID: \"d3fd153b-fb91-454a-9517-828c780a194b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:52.014924 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.014891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" Jul 08 01:42:52.027777 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.027673 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66zvg" Jul 08 01:42:52.029214 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:52.029182 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf998a88_47fd_4241_bea5_d3cac7487175.slice/crio-b2fc954eefaf86a5cb89a54e8b80c6c3aa091d5ed41c77443fbc6dd5e40beb53 WatchSource:0}: Error finding container b2fc954eefaf86a5cb89a54e8b80c6c3aa091d5ed41c77443fbc6dd5e40beb53: Status 404 returned error can't find the container with id b2fc954eefaf86a5cb89a54e8b80c6c3aa091d5ed41c77443fbc6dd5e40beb53 Jul 08 01:42:52.038198 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:52.038168 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392a0884_e501_4432_870e_bd698aaa7616.slice/crio-f9492bd4ed297524266f868da535598bd324df294aa080c6ea290a8335f2062f WatchSource:0}: Error finding container f9492bd4ed297524266f868da535598bd324df294aa080c6ea290a8335f2062f: Status 404 returned error can't find the container with id f9492bd4ed297524266f868da535598bd324df294aa080c6ea290a8335f2062f Jul 08 01:42:52.040735 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.040558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54f2j" Jul 08 01:42:52.051225 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.047748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:42:52.052020 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:52.051994 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda542d4a8_3f11_4d97_b235_71de248f8a72.slice/crio-2056ed2f23ca910ddbc121c11ba1ec5958bef06244124d4bda10f8e9ac895457 WatchSource:0}: Error finding container 2056ed2f23ca910ddbc121c11ba1ec5958bef06244124d4bda10f8e9ac895457: Status 404 returned error can't find the container with id 2056ed2f23ca910ddbc121c11ba1ec5958bef06244124d4bda10f8e9ac895457 Jul 08 01:42:52.060496 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:52.060450 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fd153b_fb91_454a_9517_828c780a194b.slice/crio-fdcfef41eb0d7a39b73aa2e0dee24f9216ac7d68f3be4808a9fbdcb565f1d230 WatchSource:0}: Error finding container fdcfef41eb0d7a39b73aa2e0dee24f9216ac7d68f3be4808a9fbdcb565f1d230: Status 404 returned error can't find the container with id fdcfef41eb0d7a39b73aa2e0dee24f9216ac7d68f3be4808a9fbdcb565f1d230 Jul 08 01:42:52.063070 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.063023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" Jul 08 01:42:52.073804 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:52.073778 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529292c4_1ccf_4ebc_a014_af5094659f0c.slice/crio-86f9615d29f754a8c243b69880a9011858a836689063ee230da0dc2eecd5b294 WatchSource:0}: Error finding container 86f9615d29f754a8c243b69880a9011858a836689063ee230da0dc2eecd5b294: Status 404 returned error can't find the container with id 86f9615d29f754a8c243b69880a9011858a836689063ee230da0dc2eecd5b294 Jul 08 01:42:52.113915 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.113837 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:52.124400 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:52.124367 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d6c6dc_5229_44fb_8744_715743709f96.slice/crio-1fe2bf06fbe6df4c2da35432fbae2e766b4c3a0f909cb7c31b938063280dfaf5 WatchSource:0}: Error finding container 1fe2bf06fbe6df4c2da35432fbae2e766b4c3a0f909cb7c31b938063280dfaf5: Status 404 returned error can't find the container with id 1fe2bf06fbe6df4c2da35432fbae2e766b4c3a0f909cb7c31b938063280dfaf5 Jul 08 01:42:52.129599 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.129512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l6grg" Jul 08 01:42:52.139715 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.139684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" Jul 08 01:42:52.151947 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.151924 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m5xj7" Jul 08 01:42:52.436892 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.436788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:52.437063 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.436895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:52.437063 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.436944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:52.437063 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:52.437019 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:52.437231 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:52.437101 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret podName:1a6bb622-0bb3-4f7a-85bd-5714b6756d82 nodeName:}" failed. No retries permitted until 2026-07-08 01:42:53.437080504 +0000 UTC m=+4.098995043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret") pod "global-pull-secret-syncer-fm4cg" (UID: "1a6bb622-0bb3-4f7a-85bd-5714b6756d82") : object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:52.437231 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:52.437105 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:52.437231 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:52.437106 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 01:42:52.437231 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:52.437195 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 01:42:52.437231 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:52.437210 2578 projected.go:196] Error preparing data for projected volume kube-api-access-ttww6 for pod openshift-network-diagnostics/network-check-target-62q7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:52.437544 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:52.437162 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs podName:85d121a8-c00e-4baf-a410-40d9ac037d1c nodeName:}" failed. No retries permitted until 2026-07-08 01:42:53.437144643 +0000 UTC m=+4.099059165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs") pod "network-metrics-daemon-852qz" (UID: "85d121a8-c00e-4baf-a410-40d9ac037d1c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:52.437544 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:52.437277 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6 podName:24490e15-c228-4567-a883-c504c7cb6975 nodeName:}" failed. No retries permitted until 2026-07-08 01:42:53.437261175 +0000 UTC m=+4.099175691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ttww6" (UniqueName: "kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6") pod "network-check-target-62q7j" (UID: "24490e15-c228-4567-a883-c504c7cb6975") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:52.764310 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.764210 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 01:37:52 +0000 UTC" deadline="2027-12-02 20:51:08.05849814 +0000 UTC" Jul 08 01:42:52.764310 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.764258 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12307h8m15.294242718s" Jul 08 01:42:52.912332 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.912287 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" event={"ID":"d3fd153b-fb91-454a-9517-828c780a194b","Type":"ContainerStarted","Data":"fdcfef41eb0d7a39b73aa2e0dee24f9216ac7d68f3be4808a9fbdcb565f1d230"} Jul 08 01:42:52.913655 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.913618 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54f2j" event={"ID":"a542d4a8-3f11-4d97-b235-71de248f8a72","Type":"ContainerStarted","Data":"2056ed2f23ca910ddbc121c11ba1ec5958bef06244124d4bda10f8e9ac895457"} Jul 08 01:42:52.915952 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.915920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66zvg" event={"ID":"392a0884-e501-4432-870e-bd698aaa7616","Type":"ContainerStarted","Data":"f9492bd4ed297524266f868da535598bd324df294aa080c6ea290a8335f2062f"} Jul 08 01:42:52.917641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.917612 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" event={"ID":"af998a88-47fd-4241-bea5-d3cac7487175","Type":"ContainerStarted","Data":"b2fc954eefaf86a5cb89a54e8b80c6c3aa091d5ed41c77443fbc6dd5e40beb53"} Jul 08 01:42:52.919209 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.919156 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7qll" event={"ID":"a5d6c6dc-5229-44fb-8744-715743709f96","Type":"ContainerStarted","Data":"1fe2bf06fbe6df4c2da35432fbae2e766b4c3a0f909cb7c31b938063280dfaf5"} Jul 08 01:42:52.920617 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.920591 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" event={"ID":"529292c4-1ccf-4ebc-a014-af5094659f0c","Type":"ContainerStarted","Data":"86f9615d29f754a8c243b69880a9011858a836689063ee230da0dc2eecd5b294"} Jul 08 01:42:52.995057 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:52.995017 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 01:42:53.310738 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:53.310680 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f036fb_7c1c_427c_a1df_0da4575540d0.slice/crio-32efb6e718f827749f388fb2cdc0866f4d3959eb1426b0ec5e1664eceb4b5996 WatchSource:0}: Error finding container 32efb6e718f827749f388fb2cdc0866f4d3959eb1426b0ec5e1664eceb4b5996: Status 404 returned error can't find the container with id 32efb6e718f827749f388fb2cdc0866f4d3959eb1426b0ec5e1664eceb4b5996 Jul 08 01:42:53.312635 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:53.312608 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061de9e3_7d37_4c96_aece_75722b0c41c8.slice/crio-917b92a079a9bee6f58e0eb5496a0f7757fbf2805886d63ea0a9c10bd80d4fca WatchSource:0}: Error finding container 917b92a079a9bee6f58e0eb5496a0f7757fbf2805886d63ea0a9c10bd80d4fca: Status 404 returned error can't find the container with id 917b92a079a9bee6f58e0eb5496a0f7757fbf2805886d63ea0a9c10bd80d4fca Jul 08 01:42:53.337416 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:42:53.337364 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05594ec0_102a_4511_ad57_fb9d94abac4d.slice/crio-c9a49436d36fde3bd2d2ffb00a6abab2bb13e8fa6137dd841b523dba8e040a89 WatchSource:0}: Error finding container c9a49436d36fde3bd2d2ffb00a6abab2bb13e8fa6137dd841b523dba8e040a89: Status 404 returned error can't find the container with id c9a49436d36fde3bd2d2ffb00a6abab2bb13e8fa6137dd841b523dba8e040a89 Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.444783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.444849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.444919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.445093 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.445162 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret podName:1a6bb622-0bb3-4f7a-85bd-5714b6756d82 nodeName:}" failed. No retries permitted until 2026-07-08 01:42:55.445140822 +0000 UTC m=+6.107055365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret") pod "global-pull-secret-syncer-fm4cg" (UID: "1a6bb622-0bb3-4f7a-85bd-5714b6756d82") : object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.445610 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.445640 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.445653 2578 projected.go:196] Error preparing data for projected volume kube-api-access-ttww6 for pod openshift-network-diagnostics/network-check-target-62q7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.445704 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6 podName:24490e15-c228-4567-a883-c504c7cb6975 nodeName:}" failed. No retries permitted until 2026-07-08 01:42:55.445687906 +0000 UTC m=+6.107602439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ttww6" (UniqueName: "kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6") pod "network-check-target-62q7j" (UID: "24490e15-c228-4567-a883-c504c7cb6975") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.445787 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:53.445862 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.445826 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs podName:85d121a8-c00e-4baf-a410-40d9ac037d1c nodeName:}" failed. No retries permitted until 2026-07-08 01:42:55.445814225 +0000 UTC m=+6.107728761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs") pod "network-metrics-daemon-852qz" (UID: "85d121a8-c00e-4baf-a410-40d9ac037d1c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:53.765033 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.764899 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 01:37:52 +0000 UTC" deadline="2027-12-23 05:27:45.079785249 +0000 UTC" Jul 08 01:42:53.765033 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.764945 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12795h44m51.314845405s" Jul 08 01:42:53.900417 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.900376 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:53.900417 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.900415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:53.900670 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.900502 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:42:53.900670 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.900641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:53.900774 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.900754 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:42:53.900870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:53.900844 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:42:53.929906 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.929846 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" event={"ID":"e64f5c6be32d9511c588a4d2d5ea69a6","Type":"ContainerStarted","Data":"d73e430fcd5fc6bc6baefb7cb0b0c12b9d9b28a496f02434f249a2b1f013ff8d"} Jul 08 01:42:53.935025 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.934978 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" event={"ID":"48f036fb-7c1c-427c-a1df-0da4575540d0","Type":"ContainerStarted","Data":"32efb6e718f827749f388fb2cdc0866f4d3959eb1426b0ec5e1664eceb4b5996"} Jul 08 01:42:53.940752 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.940723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l6grg" event={"ID":"05594ec0-102a-4511-ad57-fb9d94abac4d","Type":"ContainerStarted","Data":"c9a49436d36fde3bd2d2ffb00a6abab2bb13e8fa6137dd841b523dba8e040a89"} Jul 08 01:42:53.943464 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:53.943442 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m5xj7" event={"ID":"061de9e3-7d37-4c96-aece-75722b0c41c8","Type":"ContainerStarted","Data":"917b92a079a9bee6f58e0eb5496a0f7757fbf2805886d63ea0a9c10bd80d4fca"} Jul 08 01:42:54.961272 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:54.959577 2578 generic.go:356] "Generic (PLEG): container finished" podID="0cf5b13b5c7134c2ef58eddb3c8605ed" containerID="61bd5ec9bd814a295f773367650c9e26a598dc1b305cfc064e56e289a0699caf" exitCode=0 Jul 08 01:42:54.961272 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:54.960966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" event={"ID":"0cf5b13b5c7134c2ef58eddb3c8605ed","Type":"ContainerDied","Data":"61bd5ec9bd814a295f773367650c9e26a598dc1b305cfc064e56e289a0699caf"} Jul 08 01:42:54.977585 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:54.976864 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" podStartSLOduration=3.97683922 podStartE2EDuration="3.97683922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:42:53.947111067 +0000 UTC m=+4.609025608" watchObservedRunningTime="2026-07-08 01:42:54.97683922 +0000 UTC m=+5.638753985" Jul 08 01:42:55.467343 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:55.467241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:55.467343 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:55.467305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:55.467629 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:55.467349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:55.467629 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.467499 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:55.467629 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.467527 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:55.467629 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.467564 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 01:42:55.467629 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.467580 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret podName:1a6bb622-0bb3-4f7a-85bd-5714b6756d82 nodeName:}" failed. No retries permitted until 2026-07-08 01:42:59.467559142 +0000 UTC m=+10.129473659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret") pod "global-pull-secret-syncer-fm4cg" (UID: "1a6bb622-0bb3-4f7a-85bd-5714b6756d82") : object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:55.467629 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.467593 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 01:42:55.467629 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.467599 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs podName:85d121a8-c00e-4baf-a410-40d9ac037d1c nodeName:}" failed. No retries permitted until 2026-07-08 01:42:59.467590349 +0000 UTC m=+10.129504867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs") pod "network-metrics-daemon-852qz" (UID: "85d121a8-c00e-4baf-a410-40d9ac037d1c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:55.467629 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.467607 2578 projected.go:196] Error preparing data for projected volume kube-api-access-ttww6 for pod openshift-network-diagnostics/network-check-target-62q7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:55.468058 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.467661 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6 podName:24490e15-c228-4567-a883-c504c7cb6975 nodeName:}" failed. No retries permitted until 2026-07-08 01:42:59.46764442 +0000 UTC m=+10.129558956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ttww6" (UniqueName: "kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6") pod "network-check-target-62q7j" (UID: "24490e15-c228-4567-a883-c504c7cb6975") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:55.901772 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:55.901685 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:55.901936 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.901841 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:42:55.902278 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:55.902255 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:55.902369 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.902351 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:42:55.902764 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:55.902582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:55.902764 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:55.902691 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:42:55.967942 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:55.967903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" event={"ID":"0cf5b13b5c7134c2ef58eddb3c8605ed","Type":"ContainerStarted","Data":"bb0f0a33185fab662ce8bbf5bb053be8ed0eb3085af0d2cfe8b373f8c7338181"} Jul 08 01:42:55.971854 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:55.971823 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7qll" event={"ID":"a5d6c6dc-5229-44fb-8744-715743709f96","Type":"ContainerStarted","Data":"dcbb4586bf99b85c5c015162f79ef57a8ceeddc1fc8fbb3d726e7ad756915304"} Jul 08 01:42:56.009352 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:56.008184 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" podStartSLOduration=5.008160181 podStartE2EDuration="5.008160181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:42:55.987662565 +0000 UTC m=+6.649577106" watchObservedRunningTime="2026-07-08 01:42:56.008160181 +0000 UTC m=+6.670074723" Jul 08 01:42:57.901356 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:57.901249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:57.901845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:57.901259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:57.901845 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:57.901414 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:42:57.901845 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:57.901613 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:42:57.901845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:57.901268 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:57.902075 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:57.901882 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:42:59.504901 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:59.504848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:59.504917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:59.504948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.505060 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.505122 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret podName:1a6bb622-0bb3-4f7a-85bd-5714b6756d82 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:07.505100324 +0000 UTC m=+18.167014863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret") pod "global-pull-secret-syncer-fm4cg" (UID: "1a6bb622-0bb3-4f7a-85bd-5714b6756d82") : object "kube-system"/"original-pull-secret" not registered Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.505209 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.505226 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.505240 2578 projected.go:196] Error preparing data for projected volume kube-api-access-ttww6 for pod openshift-network-diagnostics/network-check-target-62q7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.505283 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6 podName:24490e15-c228-4567-a883-c504c7cb6975 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:07.505268427 +0000 UTC m=+18.167182958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ttww6" (UniqueName: "kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6") pod "network-check-target-62q7j" (UID: "24490e15-c228-4567-a883-c504c7cb6975") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.505352 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:59.505374 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.505388 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs podName:85d121a8-c00e-4baf-a410-40d9ac037d1c nodeName:}" failed. No retries permitted until 2026-07-08 01:43:07.505376053 +0000 UTC m=+18.167290589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs") pod "network-metrics-daemon-852qz" (UID: "85d121a8-c00e-4baf-a410-40d9ac037d1c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:42:59.610411 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:59.610374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:59.611182 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:59.611155 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:42:59.626867 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:59.626809 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l7qll" podStartSLOduration=6.967338135 podStartE2EDuration="9.626788277s" totalImagesPullingTime="2.659450142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:42:56.009317068 +0000 UTC m=+6.671231610" watchObservedRunningTime="2026-07-08 01:42:59.626788277 +0000 UTC m=+10.288702816" Jul 08 01:42:59.900587 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:59.900260 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:42:59.900587 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.900385 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:42:59.900587 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:59.900568 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:42:59.900846 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.900657 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:42:59.900846 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:42:59.900716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:42:59.900846 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:42:59.900774 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:01.900238 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:01.900207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:01.900763 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:01.900346 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:01.900763 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:01.900370 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:01.900763 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:01.900458 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:01.900942 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:01.900852 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:01.900979 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:01.900956 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:03.900231 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:03.900186 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:03.900701 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:03.900193 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:03.900701 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:03.900320 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:03.900701 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:03.900363 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:03.900701 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:03.900472 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:03.900701 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:03.900602 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:05.900647 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:05.900606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:05.901131 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:05.900608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:05.901131 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:05.900732 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:05.901131 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:05.900848 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:05.901131 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:05.900608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:05.901131 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:05.900944 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:07.564378 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:07.564341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:07.564411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:07.564441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.564512 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.564545 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.564571 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.564588 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.564592 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret podName:1a6bb622-0bb3-4f7a-85bd-5714b6756d82 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:23.564573345 +0000 UTC m=+34.226487875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret") pod "global-pull-secret-syncer-fm4cg" (UID: "1a6bb622-0bb3-4f7a-85bd-5714b6756d82") : object "kube-system"/"original-pull-secret" not registered Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.564602 2578 projected.go:196] Error preparing data for projected volume kube-api-access-ttww6 for pod openshift-network-diagnostics/network-check-target-62q7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.564612 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs podName:85d121a8-c00e-4baf-a410-40d9ac037d1c nodeName:}" failed. No retries permitted until 2026-07-08 01:43:23.56460196 +0000 UTC m=+34.226516481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs") pod "network-metrics-daemon-852qz" (UID: "85d121a8-c00e-4baf-a410-40d9ac037d1c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:43:07.564870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.564639 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6 podName:24490e15-c228-4567-a883-c504c7cb6975 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:23.564630135 +0000 UTC m=+34.226544649 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ttww6" (UniqueName: "kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6") pod "network-check-target-62q7j" (UID: "24490e15-c228-4567-a883-c504c7cb6975") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:43:07.900215 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:07.900087 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:07.900215 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:07.900103 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:07.900215 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:07.900213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:07.900215 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.900209 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:07.900571 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.900322 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:07.900571 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:07.900418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:09.901583 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:09.901532 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:09.902042 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:09.901627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:09.902042 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:09.901659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:09.902042 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:09.901700 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:09.902042 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:09.901751 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:09.902042 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:09.901820 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:11.900892 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:11.900682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:11.901493 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:11.900684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:11.901493 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:11.900972 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:11.901493 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:11.900759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:11.901493 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:11.901071 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:11.901493 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:11.901109 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:12.019532 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.019496 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54f2j" event={"ID":"a542d4a8-3f11-4d97-b235-71de248f8a72","Type":"ContainerStarted","Data":"30f714df8d4be9fd1bf5ac8420d0d60ae2168f3e6b493a2847fe67f94022090c"} Jul 08 01:43:12.021013 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.020988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66zvg" event={"ID":"392a0884-e501-4432-870e-bd698aaa7616","Type":"ContainerStarted","Data":"bed67e422c12486156b9fbff98b3ffb52f54827a4fa65aa0727367294e846752"} Jul 08 01:43:12.022622 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.022589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m5xj7" event={"ID":"061de9e3-7d37-4c96-aece-75722b0c41c8","Type":"ContainerStarted","Data":"6e0bba5f4ae68461c56298ce9132521f75a485e6e713aaf57283434d621b272b"} Jul 08 01:43:12.023976 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.023952 2578 generic.go:356] "Generic (PLEG): container finished" podID="529292c4-1ccf-4ebc-a014-af5094659f0c" containerID="aeeae3ac13e60746d982ac1695d0f1377d6ad0d729bb460799a0db3a2027540e" exitCode=0 Jul 08 01:43:12.024055 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.024037 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" event={"ID":"529292c4-1ccf-4ebc-a014-af5094659f0c","Type":"ContainerDied","Data":"aeeae3ac13e60746d982ac1695d0f1377d6ad0d729bb460799a0db3a2027540e"} Jul 08 01:43:12.026226 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.026204 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" event={"ID":"d3fd153b-fb91-454a-9517-828c780a194b","Type":"ContainerStarted","Data":"c969fb88a6c5ca1b2fd9b1b6f3727dd26cd3a0c522c6d80e59737e0ed46c4ff5"} Jul 08 01:43:12.026363 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.026279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" event={"ID":"d3fd153b-fb91-454a-9517-828c780a194b","Type":"ContainerStarted","Data":"53a8b1bbfbdb348020bd06b23f55cfe4d5ba4de68a00f427acafc5f4f6717e12"} Jul 08 01:43:12.026363 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.026332 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" event={"ID":"d3fd153b-fb91-454a-9517-828c780a194b","Type":"ContainerStarted","Data":"28e476e975098bd593b2cf0ea5db3a28021c8889649b56101b1e51ab4e05b71f"} Jul 08 01:43:12.028005 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.027968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" event={"ID":"af998a88-47fd-4241-bea5-d3cac7487175","Type":"ContainerStarted","Data":"cb658ee734c59be3a8ea52f9343d179171a696eaf0c6afdcb9bf3049200d2e25"} Jul 08 01:43:12.029594 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.029565 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" event={"ID":"48f036fb-7c1c-427c-a1df-0da4575540d0","Type":"ContainerStarted","Data":"6679e61027759851f119cb15756e9b0a31aed9deb8574cbdf79d97576665c6df"} Jul 08 01:43:12.034215 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.034161 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-54f2j" podStartSLOduration=2.5620211619999997 podStartE2EDuration="22.034143658s" totalImagesPullingTime="19.472122496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:12.033613278 +0000 UTC m=+22.695527839" watchObservedRunningTime="2026-07-08 01:43:12.034143658 +0000 UTC m=+22.696058196" Jul 08 01:43:12.052830 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.050341 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qfsjk" podStartSLOduration=2.554317576 podStartE2EDuration="22.050321131s" totalImagesPullingTime="19.496003555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:12.050144437 +0000 UTC m=+22.712058974" watchObservedRunningTime="2026-07-08 01:43:12.050321131 +0000 UTC m=+22.712235669" Jul 08 01:43:12.064419 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.064328 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-66zvg" podStartSLOduration=2.578998833 podStartE2EDuration="22.064315843s" totalImagesPullingTime="19.48531701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:12.064007866 +0000 UTC m=+22.725922404" watchObservedRunningTime="2026-07-08 01:43:12.064315843 +0000 UTC m=+22.726230379" Jul 08 01:43:12.083012 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:12.082969 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m5xj7" podStartSLOduration=3.622133979 podStartE2EDuration="22.082958755s" totalImagesPullingTime="18.460824776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:12.082649557 +0000 UTC m=+22.744564095" watchObservedRunningTime="2026-07-08 01:43:12.082958755 +0000 UTC m=+22.744873291" Jul 08 01:43:13.036258 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.036231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" event={"ID":"d3fd153b-fb91-454a-9517-828c780a194b","Type":"ContainerStarted","Data":"95a05a4c56065ba8c5e68dfa9bf6f403a03f38565ac0a0ca32e06e3e2a25dfb3"} Jul 08 01:43:13.036685 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.036356 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" event={"ID":"d3fd153b-fb91-454a-9517-828c780a194b","Type":"ContainerStarted","Data":"baea680eb7b7d23ca754e0f70b33165e1ff327db8f0d92fbb60882b5336aaa11"} Jul 08 01:43:13.036685 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.036411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" event={"ID":"d3fd153b-fb91-454a-9517-828c780a194b","Type":"ContainerStarted","Data":"ee7b71c75480b12a60a6a2e26d92ac813a83a3e15c945215cf90c8681dcb72ca"} Jul 08 01:43:13.038511 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.038463 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l6grg" event={"ID":"05594ec0-102a-4511-ad57-fb9d94abac4d","Type":"ContainerStarted","Data":"f4f8efc9d46d7c7c881286b9f18fbc131e29dd94c1dba5090b4c6357a52f461f"} Jul 08 01:43:13.056040 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.056001 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l6grg" podStartSLOduration=4.869969449 podStartE2EDuration="23.055986282s" totalImagesPullingTime="18.186016833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:13.054657006 +0000 UTC m=+23.716571544" watchObservedRunningTime="2026-07-08 01:43:13.055986282 +0000 UTC m=+23.717900819" Jul 08 01:43:13.124885 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.124848 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 01:43:13.785012 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.784915 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T01:43:13.124867796Z","UUID":"7917dd34-3731-48cc-ad0c-2298d205061d","Handler":null,"Name":"","Endpoint":""} Jul 08 01:43:13.786513 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.786489 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 01:43:13.786635 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.786529 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 01:43:13.900892 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.900858 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:13.901063 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:13.900992 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:13.901404 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.901385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:13.901522 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:13.901498 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:13.901597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:13.901569 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:13.901662 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:13.901641 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:14.043096 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:14.043016 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" event={"ID":"48f036fb-7c1c-427c-a1df-0da4575540d0","Type":"ContainerStarted","Data":"79fa47cd8a4a26d5afd2f1bd6de9bca151b153ba11731138f715246db7c5ac72"} Jul 08 01:43:15.046415 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:15.046381 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" event={"ID":"48f036fb-7c1c-427c-a1df-0da4575540d0","Type":"ContainerStarted","Data":"eeed180051337c3e4a5011483e467a23b3d9cf9f3e5da2711a1242e23856cd8f"} Jul 08 01:43:15.049268 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:15.049242 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" event={"ID":"d3fd153b-fb91-454a-9517-828c780a194b","Type":"ContainerStarted","Data":"33cacf319adf620f9c57d340369587a24483b1ddf6ff64e2bb0a137532331501"} Jul 08 01:43:15.062228 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:15.062182 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h969p" podStartSLOduration=4.628823408 podStartE2EDuration="25.062165529s" totalImagesPullingTime="20.433342121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:15.062063385 +0000 UTC m=+25.723977923" watchObservedRunningTime="2026-07-08 01:43:15.062165529 +0000 UTC m=+25.724080068" Jul 08 01:43:15.900449 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:15.900236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:15.900634 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:15.900236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:15.900634 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:15.900278 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:15.900634 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:15.900567 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:15.900811 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:15.900686 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:15.900811 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:15.900773 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:17.055344 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.055302 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" event={"ID":"529292c4-1ccf-4ebc-a014-af5094659f0c","Type":"ContainerStarted","Data":"4c0cb457528dbd93720fabf6a4442588ade1f64c36c042391ecd782583d0b47f"} Jul 08 01:43:17.058763 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.058597 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" event={"ID":"d3fd153b-fb91-454a-9517-828c780a194b","Type":"ContainerStarted","Data":"661306f184983f96689d23cb272cf7e34b12f8b46e33e661a11859cb609ee7cd"} Jul 08 01:43:17.058872 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.058823 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:43:17.058927 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.058916 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:43:17.058977 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.058971 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:43:17.074532 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.074510 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:43:17.074869 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.074855 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:43:17.198978 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.198930 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" podStartSLOduration=7.689426578 podStartE2EDuration="27.19891597s" totalImagesPullingTime="19.509489392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:17.198337313 +0000 UTC m=+27.860251851" watchObservedRunningTime="2026-07-08 01:43:17.19891597 +0000 UTC m=+27.860830506" Jul 08 01:43:17.900929 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.900858 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:17.901102 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.900865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:17.901102 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:17.900945 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:17.901102 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:17.901042 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:17.901264 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:17.901108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:17.901264 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:17.901199 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:18.062024 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:18.061991 2578 generic.go:356] "Generic (PLEG): container finished" podID="529292c4-1ccf-4ebc-a014-af5094659f0c" containerID="4c0cb457528dbd93720fabf6a4442588ade1f64c36c042391ecd782583d0b47f" exitCode=0 Jul 08 01:43:18.062430 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:18.062066 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" event={"ID":"529292c4-1ccf-4ebc-a014-af5094659f0c","Type":"ContainerDied","Data":"4c0cb457528dbd93720fabf6a4442588ade1f64c36c042391ecd782583d0b47f"} Jul 08 01:43:19.901803 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:19.901770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:19.902212 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:19.901871 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:19.902212 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:19.901894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:19.902212 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:19.901916 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:19.902212 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:19.901992 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:19.902212 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:19.902071 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:20.068216 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:20.068185 2578 generic.go:356] "Generic (PLEG): container finished" podID="529292c4-1ccf-4ebc-a014-af5094659f0c" containerID="e2d0a758d3d174a1104b58e13af7e1b3824ea6204f0a0272dd4b67c27838c0e9" exitCode=0 Jul 08 01:43:20.068364 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:20.068239 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" event={"ID":"529292c4-1ccf-4ebc-a014-af5094659f0c","Type":"ContainerDied","Data":"e2d0a758d3d174a1104b58e13af7e1b3824ea6204f0a0272dd4b67c27838c0e9"} Jul 08 01:43:21.900215 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:21.900187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:21.900215 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:21.900210 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:21.900729 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:21.900210 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:21.900729 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:21.900302 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:21.900729 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:21.900372 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:21.900729 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:21.900431 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:21.906699 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:21.906679 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:43:21.906830 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:21.906761 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 08 01:43:21.907215 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:21.907182 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l7qll" Jul 08 01:43:22.074571 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:22.074541 2578 generic.go:356] "Generic (PLEG): container finished" podID="529292c4-1ccf-4ebc-a014-af5094659f0c" containerID="bedd88c373757e27d266e61627a60dec9d4e8cb4023beac30b3cd88d19df12f6" exitCode=0 Jul 08 01:43:22.074827 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:22.074802 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" event={"ID":"529292c4-1ccf-4ebc-a014-af5094659f0c","Type":"ContainerDied","Data":"bedd88c373757e27d266e61627a60dec9d4e8cb4023beac30b3cd88d19df12f6"} Jul 08 01:43:23.594335 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:23.594300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:23.594344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:23.594400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.594472 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.594516 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.594538 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.594545 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.594562 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs podName:85d121a8-c00e-4baf-a410-40d9ac037d1c nodeName:}" failed. No retries permitted until 2026-07-08 01:43:55.594543408 +0000 UTC m=+66.256457940 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs") pod "network-metrics-daemon-852qz" (UID: "85d121a8-c00e-4baf-a410-40d9ac037d1c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.594606 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret podName:1a6bb622-0bb3-4f7a-85bd-5714b6756d82 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:55.59458773 +0000 UTC m=+66.256502263 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret") pod "global-pull-secret-syncer-fm4cg" (UID: "1a6bb622-0bb3-4f7a-85bd-5714b6756d82") : object "kube-system"/"original-pull-secret" not registered Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.594551 2578 projected.go:196] Error preparing data for projected volume kube-api-access-ttww6 for pod openshift-network-diagnostics/network-check-target-62q7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:43:23.594870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.594651 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6 podName:24490e15-c228-4567-a883-c504c7cb6975 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:55.594641736 +0000 UTC m=+66.256556283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ttww6" (UniqueName: "kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6") pod "network-check-target-62q7j" (UID: "24490e15-c228-4567-a883-c504c7cb6975") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 01:43:23.901031 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:23.900950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:23.901031 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:23.900990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:23.901249 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:23.900954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:23.901249 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.901083 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:23.901249 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.901159 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:23.901249 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:23.901216 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:25.901143 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:25.901100 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:25.901143 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:25.901134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:25.901690 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:25.901103 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:25.901690 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:25.901238 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:25.901690 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:25.901306 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:25.901690 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:25.901401 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:27.088180 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:27.088147 2578 generic.go:356] "Generic (PLEG): container finished" podID="529292c4-1ccf-4ebc-a014-af5094659f0c" containerID="c1cedf4d7b44468f27a72f4eb4755e4e13e31a70ec1186e36d4bd0a8a093795d" exitCode=0 Jul 08 01:43:27.088548 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:27.088195 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" event={"ID":"529292c4-1ccf-4ebc-a014-af5094659f0c","Type":"ContainerDied","Data":"c1cedf4d7b44468f27a72f4eb4755e4e13e31a70ec1186e36d4bd0a8a093795d"} Jul 08 01:43:27.901028 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:27.900998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:27.901189 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:27.900998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:27.901189 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:27.901146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:27.901289 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:27.901107 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:27.901289 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:27.901224 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:27.901379 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:27.901308 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:28.092698 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:28.092663 2578 generic.go:356] "Generic (PLEG): container finished" podID="529292c4-1ccf-4ebc-a014-af5094659f0c" containerID="b3c5d7b79f089380fe60e40af1c456432ba5317732caaaae62b213276e1fe40e" exitCode=0 Jul 08 01:43:28.093071 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:28.092726 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" event={"ID":"529292c4-1ccf-4ebc-a014-af5094659f0c","Type":"ContainerDied","Data":"b3c5d7b79f089380fe60e40af1c456432ba5317732caaaae62b213276e1fe40e"} Jul 08 01:43:29.098899 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:29.098718 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" event={"ID":"529292c4-1ccf-4ebc-a014-af5094659f0c","Type":"ContainerStarted","Data":"eabf9320e1c99e92335a20c6579f946fc77a1802199472a51a30f70db6c8175e"} Jul 08 01:43:29.124200 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:29.124146 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7mn7c" podStartSLOduration=8.529353122 podStartE2EDuration="39.124132161s" totalImagesPullingTime="30.594779039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:29.123791743 +0000 UTC m=+39.785706280" watchObservedRunningTime="2026-07-08 01:43:29.124132161 +0000 UTC m=+39.786046696" Jul 08 01:43:29.198687 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:29.198657 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-852qz"] Jul 08 01:43:29.198834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:29.198787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:29.198952 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:29.198929 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:29.201339 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:29.201315 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-62q7j"] Jul 08 01:43:29.201450 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:29.201417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:29.201576 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:29.201552 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:29.210858 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:29.210834 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fm4cg"] Jul 08 01:43:29.210941 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:29.210913 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:29.210985 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:29.210975 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:30.900956 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:30.900923 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:30.901442 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:30.900928 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:30.901442 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:30.901026 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:30.901442 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:30.900928 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:30.901442 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:30.901105 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:30.901442 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:30.901195 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:32.900879 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:32.900848 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:32.900879 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:32.900866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:32.901300 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:32.900939 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fm4cg" podUID="1a6bb622-0bb3-4f7a-85bd-5714b6756d82" Jul 08 01:43:32.901300 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:32.900988 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:32.901300 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:32.901061 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-852qz" podUID="85d121a8-c00e-4baf-a410-40d9ac037d1c" Jul 08 01:43:32.901300 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:32.901103 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62q7j" podUID="24490e15-c228-4567-a883-c504c7cb6975" Jul 08 01:43:33.666130 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.666098 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeReady" Jul 08 01:43:33.666299 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.666202 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 01:43:33.715378 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.715351 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tbvrq"] Jul 08 01:43:33.733453 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.733422 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-47hv5"] Jul 08 01:43:33.733603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.733584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tbvrq" Jul 08 01:43:33.736643 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.736567 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 01:43:33.736643 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.736618 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 01:43:33.736824 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.736785 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 01:43:33.737084 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.737066 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-psvfg\"" Jul 08 01:43:33.753368 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.753349 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tbvrq"] Jul 08 01:43:33.753444 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.753375 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-47hv5"] Jul 08 01:43:33.753498 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.753453 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.756818 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.756799 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 01:43:33.756947 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.756929 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 01:43:33.757018 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.756959 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 01:43:33.757300 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.757269 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 01:43:33.757829 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.757810 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wdpsc"] Jul 08 01:43:33.766014 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.765999 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wwxml\"" Jul 08 01:43:33.774522 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.774492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de36bd07-f274-429f-b46d-05bb02465482-data-volume\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.774607 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.774521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt579\" (UniqueName: \"kubernetes.io/projected/de36bd07-f274-429f-b46d-05bb02465482-kube-api-access-gt579\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.774607 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.774554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5eb9e5b4-0014-49e9-87e6-e6aea2920399-cert\") pod \"ingress-canary-tbvrq\" (UID: \"5eb9e5b4-0014-49e9-87e6-e6aea2920399\") " pod="openshift-ingress-canary/ingress-canary-tbvrq" Jul 08 01:43:33.774684 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.774635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frcss\" (UniqueName: \"kubernetes.io/projected/5eb9e5b4-0014-49e9-87e6-e6aea2920399-kube-api-access-frcss\") pod \"ingress-canary-tbvrq\" (UID: \"5eb9e5b4-0014-49e9-87e6-e6aea2920399\") " pod="openshift-ingress-canary/ingress-canary-tbvrq" Jul 08 01:43:33.774684 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.774676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de36bd07-f274-429f-b46d-05bb02465482-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.774782 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.774699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de36bd07-f274-429f-b46d-05bb02465482-crio-socket\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.774782 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.774722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de36bd07-f274-429f-b46d-05bb02465482-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.777401 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.777382 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.779906 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.779891 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 01:43:33.780277 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.780264 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 01:43:33.788377 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.788361 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v5bn9\"" Jul 08 01:43:33.789733 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.789706 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wdpsc"] Jul 08 01:43:33.875847 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.875818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de36bd07-f274-429f-b46d-05bb02465482-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.875977 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.875854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5e9319-202e-4401-8772-d2fafe6eb896-tmp-dir\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.875977 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.875883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de36bd07-f274-429f-b46d-05bb02465482-data-volume\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.875977 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.875910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt579\" (UniqueName: \"kubernetes.io/projected/de36bd07-f274-429f-b46d-05bb02465482-kube-api-access-gt579\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.875977 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.875940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5eb9e5b4-0014-49e9-87e6-e6aea2920399-cert\") pod \"ingress-canary-tbvrq\" (UID: \"5eb9e5b4-0014-49e9-87e6-e6aea2920399\") " pod="openshift-ingress-canary/ingress-canary-tbvrq" Jul 08 01:43:33.875977 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.875961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee5e9319-202e-4401-8772-d2fafe6eb896-metrics-tls\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.876163 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.875985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp4vj\" (UniqueName: \"kubernetes.io/projected/ee5e9319-202e-4401-8772-d2fafe6eb896-kube-api-access-pp4vj\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.876163 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.876009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5e9319-202e-4401-8772-d2fafe6eb896-config-volume\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.876163 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.876114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frcss\" (UniqueName: \"kubernetes.io/projected/5eb9e5b4-0014-49e9-87e6-e6aea2920399-kube-api-access-frcss\") pod \"ingress-canary-tbvrq\" (UID: \"5eb9e5b4-0014-49e9-87e6-e6aea2920399\") " pod="openshift-ingress-canary/ingress-canary-tbvrq" Jul 08 01:43:33.876310 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.876170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de36bd07-f274-429f-b46d-05bb02465482-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.876310 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.876209 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de36bd07-f274-429f-b46d-05bb02465482-data-volume\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.876310 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.876211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de36bd07-f274-429f-b46d-05bb02465482-crio-socket\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.876452 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.876356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de36bd07-f274-429f-b46d-05bb02465482-crio-socket\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.876511 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.876497 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de36bd07-f274-429f-b46d-05bb02465482-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.879846 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.879822 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5eb9e5b4-0014-49e9-87e6-e6aea2920399-cert\") pod \"ingress-canary-tbvrq\" (UID: \"5eb9e5b4-0014-49e9-87e6-e6aea2920399\") " pod="openshift-ingress-canary/ingress-canary-tbvrq" Jul 08 01:43:33.881895 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.880042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de36bd07-f274-429f-b46d-05bb02465482-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.884872 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.884851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt579\" (UniqueName: \"kubernetes.io/projected/de36bd07-f274-429f-b46d-05bb02465482-kube-api-access-gt579\") pod \"insights-runtime-extractor-47hv5\" (UID: \"de36bd07-f274-429f-b46d-05bb02465482\") " pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:33.884995 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.884976 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frcss\" (UniqueName: \"kubernetes.io/projected/5eb9e5b4-0014-49e9-87e6-e6aea2920399-kube-api-access-frcss\") pod \"ingress-canary-tbvrq\" (UID: \"5eb9e5b4-0014-49e9-87e6-e6aea2920399\") " pod="openshift-ingress-canary/ingress-canary-tbvrq" Jul 08 01:43:33.977272 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.977219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5e9319-202e-4401-8772-d2fafe6eb896-tmp-dir\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.977272 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.977253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee5e9319-202e-4401-8772-d2fafe6eb896-metrics-tls\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.977272 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.977269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp4vj\" (UniqueName: \"kubernetes.io/projected/ee5e9319-202e-4401-8772-d2fafe6eb896-kube-api-access-pp4vj\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.977763 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.977291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5e9319-202e-4401-8772-d2fafe6eb896-config-volume\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.977763 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.977552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5e9319-202e-4401-8772-d2fafe6eb896-tmp-dir\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.977887 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.977871 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5e9319-202e-4401-8772-d2fafe6eb896-config-volume\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.979345 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.979329 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee5e9319-202e-4401-8772-d2fafe6eb896-metrics-tls\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:33.985298 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:33.985276 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp4vj\" (UniqueName: \"kubernetes.io/projected/ee5e9319-202e-4401-8772-d2fafe6eb896-kube-api-access-pp4vj\") pod \"dns-default-wdpsc\" (UID: \"ee5e9319-202e-4401-8772-d2fafe6eb896\") " pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:34.043731 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.043711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tbvrq" Jul 08 01:43:34.060829 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.060804 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-47hv5" Jul 08 01:43:34.085056 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.084905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:34.245936 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.245837 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tbvrq"] Jul 08 01:43:34.248656 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.248627 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-47hv5"] Jul 08 01:43:34.252268 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:34.252245 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb9e5b4_0014_49e9_87e6_e6aea2920399.slice/crio-8f4995b6eb44a048dcb2ebea0452d5474db4857643c4bac0dd5214434bc6555f WatchSource:0}: Error finding container 8f4995b6eb44a048dcb2ebea0452d5474db4857643c4bac0dd5214434bc6555f: Status 404 returned error can't find the container with id 8f4995b6eb44a048dcb2ebea0452d5474db4857643c4bac0dd5214434bc6555f Jul 08 01:43:34.252704 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:34.252686 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde36bd07_f274_429f_b46d_05bb02465482.slice/crio-77ea2066038e1003471195e10b1a89818097c33c2b669fe6b83aeb80eee1c6f9 WatchSource:0}: Error finding container 77ea2066038e1003471195e10b1a89818097c33c2b669fe6b83aeb80eee1c6f9: Status 404 returned error can't find the container with id 77ea2066038e1003471195e10b1a89818097c33c2b669fe6b83aeb80eee1c6f9 Jul 08 01:43:34.261300 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.261280 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wdpsc"] Jul 08 01:43:34.265699 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:34.264971 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5e9319_202e_4401_8772_d2fafe6eb896.slice/crio-14af13417478003668514d79b756f7f82b4c9bc5b383fc25776c216e21b78310 WatchSource:0}: Error finding container 14af13417478003668514d79b756f7f82b4c9bc5b383fc25776c216e21b78310: Status 404 returned error can't find the container with id 14af13417478003668514d79b756f7f82b4c9bc5b383fc25776c216e21b78310 Jul 08 01:43:34.746878 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.746848 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2"] Jul 08 01:43:34.760923 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.760842 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2"] Jul 08 01:43:34.761079 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.760948 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" Jul 08 01:43:34.763405 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.763379 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 08 01:43:34.763556 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.763441 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wbz59\"" Jul 08 01:43:34.784723 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.784697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/499318ab-de32-4681-9c47-622bf9517e65-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-xkqr2\" (UID: \"499318ab-de32-4681-9c47-622bf9517e65\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" Jul 08 01:43:34.885314 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.885278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/499318ab-de32-4681-9c47-622bf9517e65-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-xkqr2\" (UID: \"499318ab-de32-4681-9c47-622bf9517e65\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" Jul 08 01:43:34.885493 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:34.885417 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 08 01:43:34.885572 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:34.885515 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/499318ab-de32-4681-9c47-622bf9517e65-tls-certificates podName:499318ab-de32-4681-9c47-622bf9517e65 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:35.38547165 +0000 UTC m=+46.047386166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/499318ab-de32-4681-9c47-622bf9517e65-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-xkqr2" (UID: "499318ab-de32-4681-9c47-622bf9517e65") : secret "prometheus-operator-admission-webhook-tls" not found Jul 08 01:43:34.901506 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.901085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:34.901506 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.901117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:34.901506 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.901319 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:34.903734 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.903713 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 01:43:34.904062 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.904040 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pkcps\"" Jul 08 01:43:34.904297 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.904240 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 01:43:34.904297 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.904286 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 01:43:34.904548 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.904531 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2x5xt\"" Jul 08 01:43:34.904623 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:34.904560 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 01:43:35.123887 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:35.123852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-47hv5" event={"ID":"de36bd07-f274-429f-b46d-05bb02465482","Type":"ContainerStarted","Data":"77ea2066038e1003471195e10b1a89818097c33c2b669fe6b83aeb80eee1c6f9"} Jul 08 01:43:35.127386 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:35.127355 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wdpsc" event={"ID":"ee5e9319-202e-4401-8772-d2fafe6eb896","Type":"ContainerStarted","Data":"14af13417478003668514d79b756f7f82b4c9bc5b383fc25776c216e21b78310"} Jul 08 01:43:35.130102 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:35.130013 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tbvrq" event={"ID":"5eb9e5b4-0014-49e9-87e6-e6aea2920399","Type":"ContainerStarted","Data":"8f4995b6eb44a048dcb2ebea0452d5474db4857643c4bac0dd5214434bc6555f"} Jul 08 01:43:35.388986 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:35.388864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/499318ab-de32-4681-9c47-622bf9517e65-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-xkqr2\" (UID: \"499318ab-de32-4681-9c47-622bf9517e65\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" Jul 08 01:43:35.389144 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:35.389031 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 08 01:43:35.389144 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:35.389111 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/499318ab-de32-4681-9c47-622bf9517e65-tls-certificates podName:499318ab-de32-4681-9c47-622bf9517e65 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:36.389086403 +0000 UTC m=+47.051000932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/499318ab-de32-4681-9c47-622bf9517e65-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-xkqr2" (UID: "499318ab-de32-4681-9c47-622bf9517e65") : secret "prometheus-operator-admission-webhook-tls" not found Jul 08 01:43:36.398513 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:36.398457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/499318ab-de32-4681-9c47-622bf9517e65-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-xkqr2\" (UID: \"499318ab-de32-4681-9c47-622bf9517e65\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" Jul 08 01:43:36.402602 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:36.402569 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/499318ab-de32-4681-9c47-622bf9517e65-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-xkqr2\" (UID: \"499318ab-de32-4681-9c47-622bf9517e65\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" Jul 08 01:43:36.573687 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:36.573650 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" Jul 08 01:43:37.465538 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.465503 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-848887767b-p9pwz"] Jul 08 01:43:37.504759 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.504725 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848887767b-p9pwz"] Jul 08 01:43:37.504934 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.504822 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.507537 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.507512 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 01:43:37.507674 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.507643 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 01:43:37.507674 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.507645 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 01:43:37.508586 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.508356 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 01:43:37.508586 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.508392 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 01:43:37.508586 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.508422 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2szn9\"" Jul 08 01:43:37.508586 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.508438 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 01:43:37.508586 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.508437 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 01:43:37.608458 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.608424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-oauth-config\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.608641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.608525 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-service-ca\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.608641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.608556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrhz\" (UniqueName: \"kubernetes.io/projected/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-kube-api-access-qqrhz\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.608641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.608590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-oauth-serving-cert\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.608776 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.608740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-config\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.608825 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.608796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-serving-cert\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.709537 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.709498 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-oauth-serving-cert\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.709722 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.709593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-config\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.709722 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.709631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-serving-cert\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.709722 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.709662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-oauth-config\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.709722 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.709695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-service-ca\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.709722 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.709718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrhz\" (UniqueName: \"kubernetes.io/projected/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-kube-api-access-qqrhz\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.710418 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.710389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-oauth-serving-cert\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.710536 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.710424 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-config\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.710666 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.710640 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-service-ca\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.712727 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.712700 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-serving-cert\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.712849 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.712809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-oauth-config\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.717704 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.717634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrhz\" (UniqueName: \"kubernetes.io/projected/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-kube-api-access-qqrhz\") pod \"console-848887767b-p9pwz\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:37.817949 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:37.817912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:40.433565 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:40.433355 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848887767b-p9pwz"] Jul 08 01:43:40.436516 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:40.436465 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2"] Jul 08 01:43:40.543152 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:40.542955 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627218e3_b587_482b_b1c3_f3a9ebe5b0d4.slice/crio-8ac297c52c4c010687d9f6999c5ecf8291399a78c15be8389f4ddcb340813fd1 WatchSource:0}: Error finding container 8ac297c52c4c010687d9f6999c5ecf8291399a78c15be8389f4ddcb340813fd1: Status 404 returned error can't find the container with id 8ac297c52c4c010687d9f6999c5ecf8291399a78c15be8389f4ddcb340813fd1 Jul 08 01:43:40.543914 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:40.543888 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499318ab_de32_4681_9c47_622bf9517e65.slice/crio-0c32cd3bb2218ddfce79e09202c1e6d4b4fabe55c849007b588236db23b1a54f WatchSource:0}: Error finding container 0c32cd3bb2218ddfce79e09202c1e6d4b4fabe55c849007b588236db23b1a54f: Status 404 returned error can't find the container with id 0c32cd3bb2218ddfce79e09202c1e6d4b4fabe55c849007b588236db23b1a54f Jul 08 01:43:41.149137 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:41.149103 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wdpsc" event={"ID":"ee5e9319-202e-4401-8772-d2fafe6eb896","Type":"ContainerStarted","Data":"e5885ce0b4279157e07615ea5d4116a65af6d9df2d84c782bdb966d5615dbb56"} Jul 08 01:43:41.149316 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:41.149194 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wdpsc" event={"ID":"ee5e9319-202e-4401-8772-d2fafe6eb896","Type":"ContainerStarted","Data":"08f06b874b6174fee26cf4d67b206b1c4e1d0062bf0e444d0d437da951928380"} Jul 08 01:43:41.149396 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:41.149335 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:41.151333 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:41.151305 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848887767b-p9pwz" event={"ID":"627218e3-b587-482b-b1c3-f3a9ebe5b0d4","Type":"ContainerStarted","Data":"8ac297c52c4c010687d9f6999c5ecf8291399a78c15be8389f4ddcb340813fd1"} Jul 08 01:43:41.154366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:41.154330 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tbvrq" event={"ID":"5eb9e5b4-0014-49e9-87e6-e6aea2920399","Type":"ContainerStarted","Data":"76f11ff8bf5a3d234e4f99627a395676d4e5a81c2dee284c77de1f34bb131f5a"} Jul 08 01:43:41.155881 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:41.155861 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-47hv5" event={"ID":"de36bd07-f274-429f-b46d-05bb02465482","Type":"ContainerStarted","Data":"18c3f8e171acf7495c36b8ec3fba480f0ae31d27e25cb840d382709d04a6d976"} Jul 08 01:43:41.157006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:41.156984 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" event={"ID":"499318ab-de32-4681-9c47-622bf9517e65","Type":"ContainerStarted","Data":"0c32cd3bb2218ddfce79e09202c1e6d4b4fabe55c849007b588236db23b1a54f"} Jul 08 01:43:41.168735 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:41.168641 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wdpsc" podStartSLOduration=2.142347449 podStartE2EDuration="8.168625302s" totalImagesPullingTime="6.026277853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:41.168298857 +0000 UTC m=+51.830213393" watchObservedRunningTime="2026-07-08 01:43:41.168625302 +0000 UTC m=+51.830539833" Jul 08 01:43:41.187810 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:41.187768 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tbvrq" podStartSLOduration=2.6175964990000002 podStartE2EDuration="8.187749202s" totalImagesPullingTime="5.570152703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:41.186922741 +0000 UTC m=+51.848837279" watchObservedRunningTime="2026-07-08 01:43:41.187749202 +0000 UTC m=+51.849663740" Jul 08 01:43:43.165323 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:43.165283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-47hv5" event={"ID":"de36bd07-f274-429f-b46d-05bb02465482","Type":"ContainerStarted","Data":"a731f93c5787e64bffab056f60fdd769a63c8526ca2f99f692cedd82bc4a3a9f"} Jul 08 01:43:43.166927 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:43.166897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" event={"ID":"499318ab-de32-4681-9c47-622bf9517e65","Type":"ContainerStarted","Data":"4ef1af54981e2ab5a6e6b86ddbe06c7db2577ee303debd3850782bc3cc69d467"} Jul 08 01:43:43.167174 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:43.167149 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" Jul 08 01:43:43.171994 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:43.171972 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" Jul 08 01:43:43.183877 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:43.183831 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-xkqr2" podStartSLOduration=7.296484924 podStartE2EDuration="9.183811396s" totalImagesPullingTime="1.887326472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:43.182180527 +0000 UTC m=+53.844095064" watchObservedRunningTime="2026-07-08 01:43:43.183811396 +0000 UTC m=+53.845725933" Jul 08 01:43:44.172398 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.172364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848887767b-p9pwz" event={"ID":"627218e3-b587-482b-b1c3-f3a9ebe5b0d4","Type":"ContainerStarted","Data":"7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727"} Jul 08 01:43:44.190713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.190674 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-848887767b-p9pwz" podStartSLOduration=3.9522130669999997 podStartE2EDuration="7.190659692s" totalImagesPullingTime="3.238446625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:44.189147237 +0000 UTC m=+54.851061785" watchObservedRunningTime="2026-07-08 01:43:44.190659692 +0000 UTC m=+54.852574228" Jul 08 01:43:44.461813 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.461724 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9c5c7f74b-s79s8"] Jul 08 01:43:44.477328 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.477299 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c5c7f74b-s79s8"] Jul 08 01:43:44.477524 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.477426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.487228 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.487199 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 01:43:44.562405 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.562366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-oauth-config\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.562578 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.562420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-serving-cert\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.562578 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.562445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-console-config\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.562578 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.562495 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-service-ca\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.562578 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.562518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-oauth-serving-cert\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.562869 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.562587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkz9\" (UniqueName: \"kubernetes.io/projected/b5a1c592-2f96-4918-847f-223fb5035a1c-kube-api-access-jtkz9\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.562869 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.562659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-trusted-ca-bundle\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.663101 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.663067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-trusted-ca-bundle\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.663101 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.663105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-oauth-config\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.663322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.663132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-serving-cert\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.663322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.663189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-console-config\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.663322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.663230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-service-ca\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.663322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.663253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-oauth-serving-cert\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.663531 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.663323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkz9\" (UniqueName: \"kubernetes.io/projected/b5a1c592-2f96-4918-847f-223fb5035a1c-kube-api-access-jtkz9\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.663975 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.663952 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-service-ca\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.664085 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.664039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-console-config\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.664142 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.664112 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-oauth-serving-cert\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.664456 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.664436 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-trusted-ca-bundle\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.668568 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.668544 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-oauth-config\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.668669 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.668636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-serving-cert\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.671413 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.671390 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkz9\" (UniqueName: \"kubernetes.io/projected/b5a1c592-2f96-4918-847f-223fb5035a1c-kube-api-access-jtkz9\") pod \"console-9c5c7f74b-s79s8\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:44.791698 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:44.791627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:45.933895 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:45.933867 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c5c7f74b-s79s8"] Jul 08 01:43:45.937123 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:45.937094 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a1c592_2f96_4918_847f_223fb5035a1c.slice/crio-70443b2f9b385ebcb9a5db6d3fd379285e6d883d993a0db520500956d7d98eab WatchSource:0}: Error finding container 70443b2f9b385ebcb9a5db6d3fd379285e6d883d993a0db520500956d7d98eab: Status 404 returned error can't find the container with id 70443b2f9b385ebcb9a5db6d3fd379285e6d883d993a0db520500956d7d98eab Jul 08 01:43:46.179552 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:46.179516 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c5c7f74b-s79s8" event={"ID":"b5a1c592-2f96-4918-847f-223fb5035a1c","Type":"ContainerStarted","Data":"f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5"} Jul 08 01:43:46.179737 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:46.179566 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c5c7f74b-s79s8" event={"ID":"b5a1c592-2f96-4918-847f-223fb5035a1c","Type":"ContainerStarted","Data":"70443b2f9b385ebcb9a5db6d3fd379285e6d883d993a0db520500956d7d98eab"} Jul 08 01:43:46.181486 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:46.181452 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-47hv5" event={"ID":"de36bd07-f274-429f-b46d-05bb02465482","Type":"ContainerStarted","Data":"9719142f2e4f50bc75557ed64c87230ad0a751b51af8d69cb8255901ad329626"} Jul 08 01:43:46.196424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:46.196300 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9c5c7f74b-s79s8" podStartSLOduration=2.196285604 podStartE2EDuration="2.196285604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:46.195759475 +0000 UTC m=+56.857674009" watchObservedRunningTime="2026-07-08 01:43:46.196285604 +0000 UTC m=+56.858200142" Jul 08 01:43:46.212023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:46.211983 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-47hv5" podStartSLOduration=1.855536099 podStartE2EDuration="13.211970642s" totalImagesPullingTime="11.356434543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:43:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:46.211578259 +0000 UTC m=+56.873492796" watchObservedRunningTime="2026-07-08 01:43:46.211970642 +0000 UTC m=+56.873885180" Jul 08 01:43:47.818527 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:47.818498 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:47.818886 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:47.818569 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:43:47.819620 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:47.819600 2578 patch_prober.go:28] interesting pod/console-848887767b-p9pwz container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 08 01:43:47.819680 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:47.819655 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-848887767b-p9pwz" podUID="627218e3-b587-482b-b1c3-f3a9ebe5b0d4" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 08 01:43:48.148949 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.148921 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9"] Jul 08 01:43:48.151077 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.151061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.153194 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.153174 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 08 01:43:48.153308 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.153211 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 01:43:48.153469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.153451 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-r2mh5\"" Jul 08 01:43:48.153643 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.153509 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 01:43:48.153763 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.153750 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 08 01:43:48.154237 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.154209 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 01:43:48.163335 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.163316 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9"] Jul 08 01:43:48.185109 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.185083 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n"] Jul 08 01:43:48.187403 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.187386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.192160 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.192141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.192273 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.192200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.192273 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.192231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfrb2\" (UniqueName: \"kubernetes.io/projected/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-kube-api-access-gfrb2\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.192420 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.192294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.202358 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.202341 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 08 01:43:48.203008 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.202994 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jtm4m\"" Jul 08 01:43:48.203736 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.203721 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 08 01:43:48.203845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.203806 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 08 01:43:48.204857 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.204842 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n"] Jul 08 01:43:48.225781 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.225448 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kg52d"] Jul 08 01:43:48.227622 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.227608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.230119 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.230092 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 01:43:48.230706 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.230687 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 01:43:48.231098 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.231065 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 01:43:48.231239 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.231222 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-658t5\"" Jul 08 01:43:48.293622 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-accelerators-collector-config\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.293778 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-wtmp\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.293778 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.293778 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/edafb75c-2df9-4915-91c2-3048cb8e7363-root\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.293778 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.293778 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfrb2\" (UniqueName: \"kubernetes.io/projected/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-kube-api-access-gfrb2\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.293778 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.293778 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dfb84f22-be4a-453b-a595-c827e9bda0dd-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.294111 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xkj\" (UniqueName: \"kubernetes.io/projected/edafb75c-2df9-4915-91c2-3048cb8e7363-kube-api-access-84xkj\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.294111 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edafb75c-2df9-4915-91c2-3048cb8e7363-sys\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.294111 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-textfile\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.294111 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jssqq\" (UniqueName: \"kubernetes.io/projected/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-api-access-jssqq\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.294111 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.293998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-tls\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.294111 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.294030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.294111 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.294053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfb84f22-be4a-453b-a595-c827e9bda0dd-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.294111 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.294077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edafb75c-2df9-4915-91c2-3048cb8e7363-metrics-client-ca\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.294423 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.294159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.294423 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.294210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.294423 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.294243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.294423 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:48.294281 2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 08 01:43:48.294423 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.294301 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.294423 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:48.294332 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-tls podName:5c0a460b-0fa2-4119-8e7b-c2cd4d792964 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:48.79431877 +0000 UTC m=+59.456233285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-lffd9" (UID: "5c0a460b-0fa2-4119-8e7b-c2cd4d792964") : secret "openshift-state-metrics-tls" not found Jul 08 01:43:48.296268 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.296252 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.302266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.302242 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfrb2\" (UniqueName: \"kubernetes.io/projected/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-kube-api-access-gfrb2\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.394826 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.394793 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/edafb75c-2df9-4915-91c2-3048cb8e7363-root\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.394956 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.394843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dfb84f22-be4a-453b-a595-c827e9bda0dd-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.394956 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.394900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/edafb75c-2df9-4915-91c2-3048cb8e7363-root\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.394956 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.394937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xkj\" (UniqueName: \"kubernetes.io/projected/edafb75c-2df9-4915-91c2-3048cb8e7363-kube-api-access-84xkj\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395067 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.394988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edafb75c-2df9-4915-91c2-3048cb8e7363-sys\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395067 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-textfile\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395067 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssqq\" (UniqueName: \"kubernetes.io/projected/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-api-access-jssqq\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.395067 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edafb75c-2df9-4915-91c2-3048cb8e7363-sys\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395067 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-tls\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395067 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfb84f22-be4a-453b-a595-c827e9bda0dd-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edafb75c-2df9-4915-91c2-3048cb8e7363-metrics-client-ca\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:48.395171 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:48.395231 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-tls podName:edafb75c-2df9-4915-91c2-3048cb8e7363 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:48.89520598 +0000 UTC m=+59.557120513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-tls") pod "node-exporter-kg52d" (UID: "edafb75c-2df9-4915-91c2-3048cb8e7363") : secret "node-exporter-tls" not found Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395235 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dfb84f22-be4a-453b-a595-c827e9bda0dd-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-accelerators-collector-config\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395313 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-textfile\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-wtmp\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395880 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395880 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395776 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-wtmp\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395985 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edafb75c-2df9-4915-91c2-3048cb8e7363-metrics-client-ca\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.395985 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.395928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfb84f22-be4a-453b-a595-c827e9bda0dd-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.396377 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.396354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.396471 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.396434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-accelerators-collector-config\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.397667 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.397647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.398003 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.397986 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.398088 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.398072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.410901 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.410839 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xkj\" (UniqueName: \"kubernetes.io/projected/edafb75c-2df9-4915-91c2-3048cb8e7363-kube-api-access-84xkj\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.411652 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.411625 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssqq\" (UniqueName: \"kubernetes.io/projected/dfb84f22-be4a-453b-a595-c827e9bda0dd-kube-api-access-jssqq\") pod \"kube-state-metrics-7c9dbb8b7d-6mx4n\" (UID: \"dfb84f22-be4a-453b-a595-c827e9bda0dd\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.495269 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.495240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" Jul 08 01:43:48.619881 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.619853 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n"] Jul 08 01:43:48.622567 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:48.622543 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb84f22_be4a_453b_a595_c827e9bda0dd.slice/crio-30a054e5ed94483c014f8ebeedcfbdb84c1be57d18fba511cb573e23043f8165 WatchSource:0}: Error finding container 30a054e5ed94483c014f8ebeedcfbdb84c1be57d18fba511cb573e23043f8165: Status 404 returned error can't find the container with id 30a054e5ed94483c014f8ebeedcfbdb84c1be57d18fba511cb573e23043f8165 Jul 08 01:43:48.798850 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.798777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:48.798979 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:48.798887 2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 08 01:43:48.798979 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:43:48.798942 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-tls podName:5c0a460b-0fa2-4119-8e7b-c2cd4d792964 nodeName:}" failed. No retries permitted until 2026-07-08 01:43:49.798929663 +0000 UTC m=+60.460844191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-lffd9" (UID: "5c0a460b-0fa2-4119-8e7b-c2cd4d792964") : secret "openshift-state-metrics-tls" not found Jul 08 01:43:48.899800 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.899772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-tls\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:48.901898 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:48.901880 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edafb75c-2df9-4915-91c2-3048cb8e7363-node-exporter-tls\") pod \"node-exporter-kg52d\" (UID: \"edafb75c-2df9-4915-91c2-3048cb8e7363\") " pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:49.075032 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:49.074962 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r544" Jul 08 01:43:49.136969 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:49.136932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kg52d" Jul 08 01:43:49.147456 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:49.147414 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedafb75c_2df9_4915_91c2_3048cb8e7363.slice/crio-9735e608bfd4aa5dd0e816703f726c345dc787e752a7fd3139c863ed6bc58d4a WatchSource:0}: Error finding container 9735e608bfd4aa5dd0e816703f726c345dc787e752a7fd3139c863ed6bc58d4a: Status 404 returned error can't find the container with id 9735e608bfd4aa5dd0e816703f726c345dc787e752a7fd3139c863ed6bc58d4a Jul 08 01:43:49.193030 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:49.192998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" event={"ID":"dfb84f22-be4a-453b-a595-c827e9bda0dd","Type":"ContainerStarted","Data":"30a054e5ed94483c014f8ebeedcfbdb84c1be57d18fba511cb573e23043f8165"} Jul 08 01:43:49.194342 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:49.194315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg52d" event={"ID":"edafb75c-2df9-4915-91c2-3048cb8e7363","Type":"ContainerStarted","Data":"9735e608bfd4aa5dd0e816703f726c345dc787e752a7fd3139c863ed6bc58d4a"} Jul 08 01:43:49.807627 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:49.807593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:49.810107 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:49.810077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c0a460b-0fa2-4119-8e7b-c2cd4d792964-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-lffd9\" (UID: \"5c0a460b-0fa2-4119-8e7b-c2cd4d792964\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:49.962522 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:49.962489 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-r2mh5\"" Jul 08 01:43:49.970983 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:49.970960 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" Jul 08 01:43:50.143470 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:50.143368 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9"] Jul 08 01:43:50.149415 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:50.149357 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0a460b_0fa2_4119_8e7b_c2cd4d792964.slice/crio-60a9d3af5ef2da9b6c2877c600dff1c9e31fb2e07494ba173bd1f9de64d89e88 WatchSource:0}: Error finding container 60a9d3af5ef2da9b6c2877c600dff1c9e31fb2e07494ba173bd1f9de64d89e88: Status 404 returned error can't find the container with id 60a9d3af5ef2da9b6c2877c600dff1c9e31fb2e07494ba173bd1f9de64d89e88 Jul 08 01:43:50.202601 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:50.202521 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" event={"ID":"dfb84f22-be4a-453b-a595-c827e9bda0dd","Type":"ContainerStarted","Data":"1fb535f447efdba8b7e235e7f8a39a4ecced6180a30fca605ea616572ddf11db"} Jul 08 01:43:50.202601 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:50.202594 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" event={"ID":"dfb84f22-be4a-453b-a595-c827e9bda0dd","Type":"ContainerStarted","Data":"ad4da2fb62b1396c97565a62dc6064c9173e75f0905a2d1451fb9b5700517720"} Jul 08 01:43:50.206938 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:50.206907 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" event={"ID":"5c0a460b-0fa2-4119-8e7b-c2cd4d792964","Type":"ContainerStarted","Data":"60a9d3af5ef2da9b6c2877c600dff1c9e31fb2e07494ba173bd1f9de64d89e88"} Jul 08 01:43:51.163072 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.162921 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wdpsc" Jul 08 01:43:51.216956 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.216914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" event={"ID":"5c0a460b-0fa2-4119-8e7b-c2cd4d792964","Type":"ContainerStarted","Data":"12826dea10af02751bef6d55d835337c3802ba923b2212dde1c06577bf266d07"} Jul 08 01:43:51.217117 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.217001 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" event={"ID":"5c0a460b-0fa2-4119-8e7b-c2cd4d792964","Type":"ContainerStarted","Data":"0933146d17228dc4bcb7f84fd754f9cd7f4025315033ba721263c5bd8fcd52e9"} Jul 08 01:43:51.219005 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.218973 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg52d" event={"ID":"edafb75c-2df9-4915-91c2-3048cb8e7363","Type":"ContainerStarted","Data":"766407354a1fc4f757e94d09914854d11e27712fb8d56f75ced4d00ece411713"} Jul 08 01:43:51.222275 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.222242 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" event={"ID":"dfb84f22-be4a-453b-a595-c827e9bda0dd","Type":"ContainerStarted","Data":"601c6adba311871b8dc7b0a75abc244c0b9ce1ab6234e0b95c0299690f6f9b55"} Jul 08 01:43:51.265323 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.265296 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9b578d7c8-kf8jr"] Jul 08 01:43:51.269728 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.269706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.272803 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.272781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 08 01:43:51.272960 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.272946 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 08 01:43:51.273022 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.272966 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ak8t1vka1trpq\"" Jul 08 01:43:51.273022 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.273009 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 08 01:43:51.273302 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.273287 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 08 01:43:51.273759 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.273736 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jp7bk\"" Jul 08 01:43:51.273915 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.273893 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 08 01:43:51.283947 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.283878 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6mx4n" podStartSLOduration=1.996962242 podStartE2EDuration="3.283859469s" totalImagesPullingTime="1.286897227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:51.282283661 +0000 UTC m=+61.944198199" watchObservedRunningTime="2026-07-08 01:43:51.283859469 +0000 UTC m=+61.945773983" Jul 08 01:43:51.287289 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.287211 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9b578d7c8-kf8jr"] Jul 08 01:43:51.321291 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.321259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-tls\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.321418 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.321346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.321497 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.321423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-grpc-tls\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.321560 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.321523 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.321560 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.321554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.321689 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.321641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.321749 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.321687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpb6q\" (UniqueName: \"kubernetes.io/projected/15c69306-2ece-463f-9af8-29ee52213869-kube-api-access-zpb6q\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.321807 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.321749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c69306-2ece-463f-9af8-29ee52213869-metrics-client-ca\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.422299 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.422262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-tls\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.422494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.422335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.422494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.422383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-grpc-tls\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.422616 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.422519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.422616 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.422554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.422713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.422611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.422713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.422650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpb6q\" (UniqueName: \"kubernetes.io/projected/15c69306-2ece-463f-9af8-29ee52213869-kube-api-access-zpb6q\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.422713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.422677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c69306-2ece-463f-9af8-29ee52213869-metrics-client-ca\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.423593 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.423543 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c69306-2ece-463f-9af8-29ee52213869-metrics-client-ca\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.425181 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.425152 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-tls\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.426187 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.426091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.426187 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.426108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-grpc-tls\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.426187 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.426144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.426396 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.426206 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.431400 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.431369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15c69306-2ece-463f-9af8-29ee52213869-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.433378 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.433357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpb6q\" (UniqueName: \"kubernetes.io/projected/15c69306-2ece-463f-9af8-29ee52213869-kube-api-access-zpb6q\") pod \"thanos-querier-9b578d7c8-kf8jr\" (UID: \"15c69306-2ece-463f-9af8-29ee52213869\") " pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.592761 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.592731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:51.741019 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:51.740986 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9b578d7c8-kf8jr"] Jul 08 01:43:51.745996 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:51.745968 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c69306_2ece_463f_9af8_29ee52213869.slice/crio-cfd249faa4e88368b0b6f053f79bb630a03ad7a722288ed51cae062187cb2c5c WatchSource:0}: Error finding container cfd249faa4e88368b0b6f053f79bb630a03ad7a722288ed51cae062187cb2c5c: Status 404 returned error can't find the container with id cfd249faa4e88368b0b6f053f79bb630a03ad7a722288ed51cae062187cb2c5c Jul 08 01:43:52.227571 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.227542 2578 generic.go:356] "Generic (PLEG): container finished" podID="edafb75c-2df9-4915-91c2-3048cb8e7363" containerID="766407354a1fc4f757e94d09914854d11e27712fb8d56f75ced4d00ece411713" exitCode=0 Jul 08 01:43:52.228016 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.227636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg52d" event={"ID":"edafb75c-2df9-4915-91c2-3048cb8e7363","Type":"ContainerDied","Data":"766407354a1fc4f757e94d09914854d11e27712fb8d56f75ced4d00ece411713"} Jul 08 01:43:52.228858 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.228836 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" event={"ID":"15c69306-2ece-463f-9af8-29ee52213869","Type":"ContainerStarted","Data":"cfd249faa4e88368b0b6f053f79bb630a03ad7a722288ed51cae062187cb2c5c"} Jul 08 01:43:52.230856 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.230826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" event={"ID":"5c0a460b-0fa2-4119-8e7b-c2cd4d792964","Type":"ContainerStarted","Data":"41e88de41b6c8e87534358d0b0dcc4a299c6409867f0d0977faf1debf3d41dbd"} Jul 08 01:43:52.261939 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.261899 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-lffd9" podStartSLOduration=3.043008032 podStartE2EDuration="4.261880358s" totalImagesPullingTime="1.218872326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:52.260877047 +0000 UTC m=+62.922791582" watchObservedRunningTime="2026-07-08 01:43:52.261880358 +0000 UTC m=+62.923794897" Jul 08 01:43:52.530591 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.530509 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6845648df-vv42s"] Jul 08 01:43:52.532948 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.532927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.535254 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.535230 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 08 01:43:52.535554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.535380 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-79bll3ih3jq2k\"" Jul 08 01:43:52.535554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.535426 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-z9bhr\"" Jul 08 01:43:52.535554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.535374 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 08 01:43:52.535554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.535458 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 08 01:43:52.535554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.535549 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 08 01:43:52.544959 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.544937 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6845648df-vv42s"] Jul 08 01:43:52.630770 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.630742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62sml\" (UniqueName: \"kubernetes.io/projected/63711f4d-76e8-4c1e-9666-5b7a5e112489-kube-api-access-62sml\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.630902 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.630790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/63711f4d-76e8-4c1e-9666-5b7a5e112489-secret-metrics-server-client-certs\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.630902 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.630825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/63711f4d-76e8-4c1e-9666-5b7a5e112489-client-ca-bundle\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.630902 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.630858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63711f4d-76e8-4c1e-9666-5b7a5e112489-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.630902 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.630874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/63711f4d-76e8-4c1e-9666-5b7a5e112489-audit-log\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.631075 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.630926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/63711f4d-76e8-4c1e-9666-5b7a5e112489-metrics-server-audit-profiles\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.631075 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.630977 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/63711f4d-76e8-4c1e-9666-5b7a5e112489-secret-metrics-server-tls\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.732154 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.732115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/63711f4d-76e8-4c1e-9666-5b7a5e112489-secret-metrics-server-client-certs\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.732322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.732191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/63711f4d-76e8-4c1e-9666-5b7a5e112489-client-ca-bundle\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.732322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.732246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63711f4d-76e8-4c1e-9666-5b7a5e112489-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.732322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.732270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/63711f4d-76e8-4c1e-9666-5b7a5e112489-audit-log\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.732322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.732302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/63711f4d-76e8-4c1e-9666-5b7a5e112489-metrics-server-audit-profiles\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.732566 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.732328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/63711f4d-76e8-4c1e-9666-5b7a5e112489-secret-metrics-server-tls\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.732566 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.732372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62sml\" (UniqueName: \"kubernetes.io/projected/63711f4d-76e8-4c1e-9666-5b7a5e112489-kube-api-access-62sml\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.732791 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.732694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/63711f4d-76e8-4c1e-9666-5b7a5e112489-audit-log\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.733335 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.733305 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63711f4d-76e8-4c1e-9666-5b7a5e112489-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.733623 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.733602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/63711f4d-76e8-4c1e-9666-5b7a5e112489-metrics-server-audit-profiles\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.735117 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.735094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/63711f4d-76e8-4c1e-9666-5b7a5e112489-secret-metrics-server-tls\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.735212 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.735120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/63711f4d-76e8-4c1e-9666-5b7a5e112489-secret-metrics-server-client-certs\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.735268 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.735229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/63711f4d-76e8-4c1e-9666-5b7a5e112489-client-ca-bundle\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.740705 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.740686 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62sml\" (UniqueName: \"kubernetes.io/projected/63711f4d-76e8-4c1e-9666-5b7a5e112489-kube-api-access-62sml\") pod \"metrics-server-6845648df-vv42s\" (UID: \"63711f4d-76e8-4c1e-9666-5b7a5e112489\") " pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.844353 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.844321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:43:52.990646 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:52.990611 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6845648df-vv42s"] Jul 08 01:43:52.993603 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:52.993576 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63711f4d_76e8_4c1e_9666_5b7a5e112489.slice/crio-2992800350a72bf8411533a1820873a31deae5ce027922e7bbc00be9b62eb415 WatchSource:0}: Error finding container 2992800350a72bf8411533a1820873a31deae5ce027922e7bbc00be9b62eb415: Status 404 returned error can't find the container with id 2992800350a72bf8411533a1820873a31deae5ce027922e7bbc00be9b62eb415 Jul 08 01:43:53.235761 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.235679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6845648df-vv42s" event={"ID":"63711f4d-76e8-4c1e-9666-5b7a5e112489","Type":"ContainerStarted","Data":"2992800350a72bf8411533a1820873a31deae5ce027922e7bbc00be9b62eb415"} Jul 08 01:43:53.237701 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.237676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg52d" event={"ID":"edafb75c-2df9-4915-91c2-3048cb8e7363","Type":"ContainerStarted","Data":"4f53adbb21054bfe7973ed5b2e0e9229b79cdec074d638a7f9d6aa74910a567e"} Jul 08 01:43:53.237804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.237718 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg52d" event={"ID":"edafb75c-2df9-4915-91c2-3048cb8e7363","Type":"ContainerStarted","Data":"cbc837e62c475a7beb52ea43f42c55cd2e6f3ead6dc884958583fb8dcf3612b5"} Jul 08 01:43:53.261072 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.260858 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kg52d" podStartSLOduration=3.314450849 podStartE2EDuration="5.260839663s" totalImagesPullingTime="1.827436792s" totalInitContainerRuntime="118.952022ms" isStatefulPod=true podCreationTimestamp="2026-07-08 01:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:53.260005833 +0000 UTC m=+63.921920382" watchObservedRunningTime="2026-07-08 01:43:53.260839663 +0000 UTC m=+63.922754201" Jul 08 01:43:53.491717 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.491626 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848887767b-p9pwz"] Jul 08 01:43:53.527300 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.527274 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-668457bc49-zfp8t"] Jul 08 01:43:53.529960 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.529941 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.541771 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.541748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668457bc49-zfp8t"] Jul 08 01:43:53.641272 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.641241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-trusted-ca-bundle\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.641272 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.641276 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46n7j\" (UniqueName: \"kubernetes.io/projected/679a371f-a778-476a-94b8-2ae02a3d5605-kube-api-access-46n7j\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.641466 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.641323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-oauth-config\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.641466 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.641359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-service-ca\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.641466 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.641381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-console-config\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.641466 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.641417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-serving-cert\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.641466 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.641434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-oauth-serving-cert\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.742386 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.742298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-console-config\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.742386 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.742378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-serving-cert\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.742641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.742405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-oauth-serving-cert\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.742641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.742473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-trusted-ca-bundle\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.742641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.742523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46n7j\" (UniqueName: \"kubernetes.io/projected/679a371f-a778-476a-94b8-2ae02a3d5605-kube-api-access-46n7j\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.742641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.742569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-oauth-config\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.742641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.742625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-service-ca\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.743237 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.743189 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-console-config\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.743343 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.743315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-oauth-serving-cert\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.743832 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.743806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-service-ca\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.743982 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.743959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-trusted-ca-bundle\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.745194 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.745172 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-serving-cert\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.745398 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.745359 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-oauth-config\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.750713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.750693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46n7j\" (UniqueName: \"kubernetes.io/projected/679a371f-a778-476a-94b8-2ae02a3d5605-kube-api-access-46n7j\") pod \"console-668457bc49-zfp8t\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:53.838332 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:53.838297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:43:54.335589 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.335392 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668457bc49-zfp8t"] Jul 08 01:43:54.339085 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:54.339052 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679a371f_a778_476a_94b8_2ae02a3d5605.slice/crio-91d3aec1c3ab2a4a85fb6cff8da17c295fdfb7fddbb48279e91f72a2f7cfc9e4 WatchSource:0}: Error finding container 91d3aec1c3ab2a4a85fb6cff8da17c295fdfb7fddbb48279e91f72a2f7cfc9e4: Status 404 returned error can't find the container with id 91d3aec1c3ab2a4a85fb6cff8da17c295fdfb7fddbb48279e91f72a2f7cfc9e4 Jul 08 01:43:54.363145 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.362667 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 01:43:54.366207 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.366188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.369114 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.369071 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 01:43:54.369731 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.369404 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 01:43:54.369731 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.369426 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 01:43:54.369731 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.369444 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 01:43:54.369731 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.369470 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 01:43:54.370099 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.369774 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xvxlq\"" Jul 08 01:43:54.370099 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.369916 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 01:43:54.370197 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.370151 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 01:43:54.370251 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.370151 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 01:43:54.372246 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.372227 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-uq3tj0ovietm\"" Jul 08 01:43:54.372661 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.372644 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 01:43:54.376033 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.376015 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 01:43:54.376628 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.376355 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 01:43:54.376628 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.376393 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 01:43:54.376628 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.376413 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 01:43:54.378215 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.378198 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 01:43:54.391103 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.391070 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 01:43:54.453066 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453162 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453162 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453234 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453234 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453234 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453328 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453328 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453390 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453390 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453390 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453486 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453486 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453486 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ffm\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-kube-api-access-p2ffm\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453699 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453699 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.453699 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.453640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554311 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554521 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554521 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554521 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ffm\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-kube-api-access-p2ffm\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554521 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554521 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554521 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554521 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.554864 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.556130 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.556130 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.554926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.556130 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.555673 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.556130 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.555887 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.556130 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.556077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.556407 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.556126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.557831 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.557803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.558528 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.558500 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.558624 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.558537 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.562000 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.558993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.562000 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.559549 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.562000 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.560756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.562000 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.560874 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.562000 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.561449 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.562000 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.561529 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.562000 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.561974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.562403 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.562387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.563073 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.563050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.563148 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.563055 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.563716 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.563695 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ffm\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-kube-api-access-p2ffm\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.563969 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.563943 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.564647 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.564623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.687451 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.687414 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:43:54.792367 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.792334 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:54.792559 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.792425 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:54.798399 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.798378 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:54.962534 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:54.962427 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 01:43:54.967389 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:54.967360 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e45bebc_23b0_4b1b_9bb1_2964d441c271.slice/crio-d742f5e1c9cbf2e6351031b9a8069cc1c6c6adcbdeb003ccbf210c2b806080b9 WatchSource:0}: Error finding container d742f5e1c9cbf2e6351031b9a8069cc1c6c6adcbdeb003ccbf210c2b806080b9: Status 404 returned error can't find the container with id d742f5e1c9cbf2e6351031b9a8069cc1c6c6adcbdeb003ccbf210c2b806080b9 Jul 08 01:43:55.262907 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.262877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6845648df-vv42s" event={"ID":"63711f4d-76e8-4c1e-9666-5b7a5e112489","Type":"ContainerStarted","Data":"6e8a5aec2ae8856fd81508eaf3827d8b30dfed20cf883ecb2a97d09d07bf34d4"} Jul 08 01:43:55.265422 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.265395 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" event={"ID":"15c69306-2ece-463f-9af8-29ee52213869","Type":"ContainerStarted","Data":"45593e38cdb2d80cf9e95bcd268ad87f4c5bdf143d9a80d7b99dda797435641f"} Jul 08 01:43:55.265543 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.265463 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" event={"ID":"15c69306-2ece-463f-9af8-29ee52213869","Type":"ContainerStarted","Data":"6477c302b252cbb82d1f721c13e3ca94562d9ce016b128b9520e240fa3261e26"} Jul 08 01:43:55.265543 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.265524 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" event={"ID":"15c69306-2ece-463f-9af8-29ee52213869","Type":"ContainerStarted","Data":"e0211bb1d29e3c3b4e2dc0d025e1f1a6aa5509562bd553fd175bb58aa8496ed6"} Jul 08 01:43:55.265646 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.265557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" event={"ID":"15c69306-2ece-463f-9af8-29ee52213869","Type":"ContainerStarted","Data":"f45c747672847b7f1d344b0fcf8baddd49d508c80347c1c15a1a0e1dd04d4b15"} Jul 08 01:43:55.266661 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.266642 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerStarted","Data":"d742f5e1c9cbf2e6351031b9a8069cc1c6c6adcbdeb003ccbf210c2b806080b9"} Jul 08 01:43:55.268266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.268242 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668457bc49-zfp8t" event={"ID":"679a371f-a778-476a-94b8-2ae02a3d5605","Type":"ContainerStarted","Data":"3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6"} Jul 08 01:43:55.268345 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.268284 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668457bc49-zfp8t" event={"ID":"679a371f-a778-476a-94b8-2ae02a3d5605","Type":"ContainerStarted","Data":"91d3aec1c3ab2a4a85fb6cff8da17c295fdfb7fddbb48279e91f72a2f7cfc9e4"} Jul 08 01:43:55.273444 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.273425 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:43:55.298015 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.297966 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6845648df-vv42s" podStartSLOduration=1.449902904 podStartE2EDuration="3.297945773s" totalImagesPullingTime="1.848042869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:55.296682622 +0000 UTC m=+65.958597161" watchObservedRunningTime="2026-07-08 01:43:55.297945773 +0000 UTC m=+65.959860311" Jul 08 01:43:55.332963 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.332909 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-668457bc49-zfp8t" podStartSLOduration=2.332887477 podStartE2EDuration="2.332887477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:55.332382174 +0000 UTC m=+65.994296712" watchObservedRunningTime="2026-07-08 01:43:55.332887477 +0000 UTC m=+65.994802014" Jul 08 01:43:55.667491 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.667442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:55.667908 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.667562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:55.667908 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.667597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:55.669912 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.669887 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 01:43:55.669996 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.669933 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 01:43:55.669996 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.669957 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 01:43:55.680203 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.680166 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 01:43:55.683243 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.680681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a6bb622-0bb3-4f7a-85bd-5714b6756d82-original-pull-secret\") pod \"global-pull-secret-syncer-fm4cg\" (UID: \"1a6bb622-0bb3-4f7a-85bd-5714b6756d82\") " pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:55.684878 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.684842 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d121a8-c00e-4baf-a410-40d9ac037d1c-metrics-certs\") pod \"network-metrics-daemon-852qz\" (UID: \"85d121a8-c00e-4baf-a410-40d9ac037d1c\") " pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:55.691708 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.691684 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttww6\" (UniqueName: \"kubernetes.io/projected/24490e15-c228-4567-a883-c504c7cb6975-kube-api-access-ttww6\") pod \"network-check-target-62q7j\" (UID: \"24490e15-c228-4567-a883-c504c7cb6975\") " pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:55.916716 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.916683 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pkcps\"" Jul 08 01:43:55.924535 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.924461 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-852qz" Jul 08 01:43:55.924698 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.924679 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2x5xt\"" Jul 08 01:43:55.930058 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.930044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fm4cg" Jul 08 01:43:55.933220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:55.933208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:43:56.082695 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.082626 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fm4cg"] Jul 08 01:43:56.089017 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:56.088990 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6bb622_0bb3_4f7a_85bd_5714b6756d82.slice/crio-3f230cac2cdfe86ff14fbeddd04385d948875524468d60071c9936d1b4753181 WatchSource:0}: Error finding container 3f230cac2cdfe86ff14fbeddd04385d948875524468d60071c9936d1b4753181: Status 404 returned error can't find the container with id 3f230cac2cdfe86ff14fbeddd04385d948875524468d60071c9936d1b4753181 Jul 08 01:43:56.274832 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.274740 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" event={"ID":"15c69306-2ece-463f-9af8-29ee52213869","Type":"ContainerStarted","Data":"ea609005f52df689d608b0ba3356e41c30b1b8b8c7b5b315a5ba6b9100fc3e78"} Jul 08 01:43:56.274832 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.274812 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" event={"ID":"15c69306-2ece-463f-9af8-29ee52213869","Type":"ContainerStarted","Data":"a44936362782aabc1f29c954a008343b178245cfa95cbfb72fc228acd390c495"} Jul 08 01:43:56.275017 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.274888 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:43:56.276320 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.276295 2578 generic.go:356] "Generic (PLEG): container finished" podID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerID="f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b" exitCode=0 Jul 08 01:43:56.276454 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.276364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerDied","Data":"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b"} Jul 08 01:43:56.277717 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.277693 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fm4cg" event={"ID":"1a6bb622-0bb3-4f7a-85bd-5714b6756d82","Type":"ContainerStarted","Data":"3f230cac2cdfe86ff14fbeddd04385d948875524468d60071c9936d1b4753181"} Jul 08 01:43:56.298003 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.297964 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" podStartSLOduration=2.127102304 podStartE2EDuration="5.297950628s" totalImagesPullingTime="3.170848324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:56.29695951 +0000 UTC m=+66.958874048" watchObservedRunningTime="2026-07-08 01:43:56.297950628 +0000 UTC m=+66.959865164" Jul 08 01:43:56.304908 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.304885 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-852qz"] Jul 08 01:43:56.306250 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:56.306232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-62q7j"] Jul 08 01:43:56.308462 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:56.308430 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24490e15_c228_4567_a883_c504c7cb6975.slice/crio-710c9fb0a39c17b7285f23d230db05fdccd7c4183b28230723447f84d4b4f78a WatchSource:0}: Error finding container 710c9fb0a39c17b7285f23d230db05fdccd7c4183b28230723447f84d4b4f78a: Status 404 returned error can't find the container with id 710c9fb0a39c17b7285f23d230db05fdccd7c4183b28230723447f84d4b4f78a Jul 08 01:43:56.309106 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:43:56.309017 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d121a8_c00e_4baf_a410_40d9ac037d1c.slice/crio-5986d39a7ef78ac6faced39ff841103e50b1cf07f7921cc793a03c27efd35187 WatchSource:0}: Error finding container 5986d39a7ef78ac6faced39ff841103e50b1cf07f7921cc793a03c27efd35187: Status 404 returned error can't find the container with id 5986d39a7ef78ac6faced39ff841103e50b1cf07f7921cc793a03c27efd35187 Jul 08 01:43:57.284439 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:57.284396 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-62q7j" event={"ID":"24490e15-c228-4567-a883-c504c7cb6975","Type":"ContainerStarted","Data":"710c9fb0a39c17b7285f23d230db05fdccd7c4183b28230723447f84d4b4f78a"} Jul 08 01:43:57.286972 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:57.286911 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-852qz" event={"ID":"85d121a8-c00e-4baf-a410-40d9ac037d1c","Type":"ContainerStarted","Data":"5986d39a7ef78ac6faced39ff841103e50b1cf07f7921cc793a03c27efd35187"} Jul 08 01:43:58.297701 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:58.297666 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-852qz" event={"ID":"85d121a8-c00e-4baf-a410-40d9ac037d1c","Type":"ContainerStarted","Data":"0975db2597b88f7855f20b9ce3262d05427d936c5938a0b3ad3446592c39632d"} Jul 08 01:43:58.298019 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:58.297732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-852qz" event={"ID":"85d121a8-c00e-4baf-a410-40d9ac037d1c","Type":"ContainerStarted","Data":"4371ab691dae51b7fc1cde6d3b4cc5b5827d930f3b00a93f8ca873caf96d22a1"} Jul 08 01:43:58.316132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:43:58.315890 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-852qz" podStartSLOduration=66.742116717 podStartE2EDuration="1m8.315870335s" totalImagesPullingTime="1.573753618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:43:58.313888669 +0000 UTC m=+68.975803208" watchObservedRunningTime="2026-07-08 01:43:58.315870335 +0000 UTC m=+68.977784872" Jul 08 01:44:02.295700 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:02.295669 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9b578d7c8-kf8jr" Jul 08 01:44:02.319368 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:02.319306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerStarted","Data":"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1"} Jul 08 01:44:02.319501 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:02.319383 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerStarted","Data":"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b"} Jul 08 01:44:02.320881 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:02.320802 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fm4cg" event={"ID":"1a6bb622-0bb3-4f7a-85bd-5714b6756d82","Type":"ContainerStarted","Data":"cc8b218568163bbc3611ee60b6fc9d15e79f0875c0f92f5f22413eff3908e07f"} Jul 08 01:44:02.323156 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:02.323132 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-62q7j" event={"ID":"24490e15-c228-4567-a883-c504c7cb6975","Type":"ContainerStarted","Data":"8455f00a7076a264fbd2e784f460a26f188e572c7cef71f66e2bfc352cdae568"} Jul 08 01:44:02.323705 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:02.323683 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:44:02.439103 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:02.438991 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fm4cg" podStartSLOduration=66.37898172 podStartE2EDuration="1m12.438975071s" totalImagesPullingTime="6.059993351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:44:02.437288652 +0000 UTC m=+73.099203191" watchObservedRunningTime="2026-07-08 01:44:02.438975071 +0000 UTC m=+73.100889607" Jul 08 01:44:03.330758 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:03.330719 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerStarted","Data":"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341"} Jul 08 01:44:03.331198 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:03.330786 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerStarted","Data":"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e"} Jul 08 01:44:03.331198 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:03.330825 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerStarted","Data":"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067"} Jul 08 01:44:03.331198 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:03.330863 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerStarted","Data":"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829"} Jul 08 01:44:03.375555 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:03.375503 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-62q7j" podStartSLOduration=67.547127009 podStartE2EDuration="1m13.375467318s" totalImagesPullingTime="5.828340309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:44:02.476046191 +0000 UTC m=+73.137960729" watchObservedRunningTime="2026-07-08 01:44:03.375467318 +0000 UTC m=+74.037381857" Jul 08 01:44:03.375937 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:03.375897 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.465509956 podStartE2EDuration="9.37588453s" totalImagesPullingTime="6.891104544s" totalInitContainerRuntime="19.27003ms" isStatefulPod=false podCreationTimestamp="2026-07-08 01:43:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:44:03.37327119 +0000 UTC m=+74.035185737" watchObservedRunningTime="2026-07-08 01:44:03.37588453 +0000 UTC m=+74.037799067" Jul 08 01:44:03.838831 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:03.838797 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:44:03.839007 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:03.838857 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:44:03.843795 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:03.843774 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:44:04.343464 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:04.343435 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:44:04.439834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:04.439802 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9c5c7f74b-s79s8"] Jul 08 01:44:04.688352 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:04.688281 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:44:12.845471 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:12.845356 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:44:12.845471 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:12.845436 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:44:18.511067 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.511017 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-848887767b-p9pwz" podUID="627218e3-b587-482b-b1c3-f3a9ebe5b0d4" containerName="console" containerID="cri-o://7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727" gracePeriod=15 Jul 08 01:44:18.760949 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.760930 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:44:18.862311 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862287 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqrhz\" (UniqueName: \"kubernetes.io/projected/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-kube-api-access-qqrhz\") pod \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " Jul 08 01:44:18.862442 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862317 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-config\") pod \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " Jul 08 01:44:18.862442 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862360 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-oauth-config\") pod \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " Jul 08 01:44:18.862556 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862526 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-service-ca\") pod \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " Jul 08 01:44:18.862603 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862574 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-serving-cert\") pod \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " Jul 08 01:44:18.862668 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862654 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-oauth-serving-cert\") pod \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\" (UID: \"627218e3-b587-482b-b1c3-f3a9ebe5b0d4\") " Jul 08 01:44:18.862750 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862722 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-config" (OuterVolumeSpecName: "console-config") pod "627218e3-b587-482b-b1c3-f3a9ebe5b0d4" (UID: "627218e3-b587-482b-b1c3-f3a9ebe5b0d4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:44:18.862905 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862875 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-service-ca" (OuterVolumeSpecName: "service-ca") pod "627218e3-b587-482b-b1c3-f3a9ebe5b0d4" (UID: "627218e3-b587-482b-b1c3-f3a9ebe5b0d4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:44:18.862975 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862961 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-service-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:18.863022 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862982 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:18.863022 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.862987 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "627218e3-b587-482b-b1c3-f3a9ebe5b0d4" (UID: "627218e3-b587-482b-b1c3-f3a9ebe5b0d4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:44:18.864655 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.864635 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "627218e3-b587-482b-b1c3-f3a9ebe5b0d4" (UID: "627218e3-b587-482b-b1c3-f3a9ebe5b0d4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:44:18.864756 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.864725 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-kube-api-access-qqrhz" (OuterVolumeSpecName: "kube-api-access-qqrhz") pod "627218e3-b587-482b-b1c3-f3a9ebe5b0d4" (UID: "627218e3-b587-482b-b1c3-f3a9ebe5b0d4"). InnerVolumeSpecName "kube-api-access-qqrhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:44:18.864756 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.864734 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "627218e3-b587-482b-b1c3-f3a9ebe5b0d4" (UID: "627218e3-b587-482b-b1c3-f3a9ebe5b0d4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:44:18.963668 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.963640 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-oauth-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:18.963668 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.963663 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqrhz\" (UniqueName: \"kubernetes.io/projected/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-kube-api-access-qqrhz\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:18.963668 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.963674 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-oauth-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:18.963860 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:18.963684 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/627218e3-b587-482b-b1c3-f3a9ebe5b0d4-console-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:19.399072 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.399030 2578 generic.go:356] "Generic (PLEG): container finished" podID="627218e3-b587-482b-b1c3-f3a9ebe5b0d4" containerID="7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727" exitCode=0 Jul 08 01:44:19.399275 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.399139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848887767b-p9pwz" event={"ID":"627218e3-b587-482b-b1c3-f3a9ebe5b0d4","Type":"ContainerDied","Data":"7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727"} Jul 08 01:44:19.399275 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.399225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848887767b-p9pwz" event={"ID":"627218e3-b587-482b-b1c3-f3a9ebe5b0d4","Type":"ContainerDied","Data":"8ac297c52c4c010687d9f6999c5ecf8291399a78c15be8389f4ddcb340813fd1"} Jul 08 01:44:19.399275 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.399258 2578 scope.go:117] "RemoveContainer" containerID="7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727" Jul 08 01:44:19.399502 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.399451 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848887767b-p9pwz" Jul 08 01:44:19.409227 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.409204 2578 scope.go:117] "RemoveContainer" containerID="7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727" Jul 08 01:44:19.409669 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:44:19.409636 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727\": container with ID starting with 7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727 not found: ID does not exist" containerID="7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727" Jul 08 01:44:19.409751 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.409674 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727"} err="failed to get container status \"7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727\": rpc error: code = NotFound desc = could not find container \"7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727\": container with ID starting with 7b5ec2efc41eff23e7987dbfc9b9e5f8e1059fa591bd4990471a627f5fab4727 not found: ID does not exist" Jul 08 01:44:19.424224 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.424202 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848887767b-p9pwz"] Jul 08 01:44:19.431692 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.431671 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-848887767b-p9pwz"] Jul 08 01:44:19.905565 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:19.905537 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="627218e3-b587-482b-b1c3-f3a9ebe5b0d4" path="/var/lib/kubelet/pods/627218e3-b587-482b-b1c3-f3a9ebe5b0d4/volumes" Jul 08 01:44:29.466345 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.466307 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-9c5c7f74b-s79s8" podUID="b5a1c592-2f96-4918-847f-223fb5035a1c" containerName="console" containerID="cri-o://f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5" gracePeriod=15 Jul 08 01:44:29.737907 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.737843 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:44:29.847730 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.847689 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-trusted-ca-bundle\") pod \"b5a1c592-2f96-4918-847f-223fb5035a1c\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " Jul 08 01:44:29.847904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.847789 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-serving-cert\") pod \"b5a1c592-2f96-4918-847f-223fb5035a1c\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " Jul 08 01:44:29.847904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.847815 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-service-ca\") pod \"b5a1c592-2f96-4918-847f-223fb5035a1c\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " Jul 08 01:44:29.847904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.847862 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-oauth-config\") pod \"b5a1c592-2f96-4918-847f-223fb5035a1c\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " Jul 08 01:44:29.848036 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.847983 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-oauth-serving-cert\") pod \"b5a1c592-2f96-4918-847f-223fb5035a1c\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " Jul 08 01:44:29.848092 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.848050 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtkz9\" (UniqueName: \"kubernetes.io/projected/b5a1c592-2f96-4918-847f-223fb5035a1c-kube-api-access-jtkz9\") pod \"b5a1c592-2f96-4918-847f-223fb5035a1c\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " Jul 08 01:44:29.848092 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.848083 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-console-config\") pod \"b5a1c592-2f96-4918-847f-223fb5035a1c\" (UID: \"b5a1c592-2f96-4918-847f-223fb5035a1c\") " Jul 08 01:44:29.848198 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.848119 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b5a1c592-2f96-4918-847f-223fb5035a1c" (UID: "b5a1c592-2f96-4918-847f-223fb5035a1c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:44:29.848258 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.848215 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-service-ca" (OuterVolumeSpecName: "service-ca") pod "b5a1c592-2f96-4918-847f-223fb5035a1c" (UID: "b5a1c592-2f96-4918-847f-223fb5035a1c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:44:29.848372 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.848347 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-trusted-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:29.848516 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.848463 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-service-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:29.848516 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.848428 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b5a1c592-2f96-4918-847f-223fb5035a1c" (UID: "b5a1c592-2f96-4918-847f-223fb5035a1c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:44:29.848810 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.848535 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-console-config" (OuterVolumeSpecName: "console-config") pod "b5a1c592-2f96-4918-847f-223fb5035a1c" (UID: "b5a1c592-2f96-4918-847f-223fb5035a1c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:44:29.850213 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.850192 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b5a1c592-2f96-4918-847f-223fb5035a1c" (UID: "b5a1c592-2f96-4918-847f-223fb5035a1c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:44:29.850309 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.850238 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a1c592-2f96-4918-847f-223fb5035a1c-kube-api-access-jtkz9" (OuterVolumeSpecName: "kube-api-access-jtkz9") pod "b5a1c592-2f96-4918-847f-223fb5035a1c" (UID: "b5a1c592-2f96-4918-847f-223fb5035a1c"). InnerVolumeSpecName "kube-api-access-jtkz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:44:29.850309 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.850278 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b5a1c592-2f96-4918-847f-223fb5035a1c" (UID: "b5a1c592-2f96-4918-847f-223fb5035a1c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:44:29.949073 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.949047 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtkz9\" (UniqueName: \"kubernetes.io/projected/b5a1c592-2f96-4918-847f-223fb5035a1c-kube-api-access-jtkz9\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:29.949073 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.949068 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-console-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:29.949197 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.949077 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:29.949197 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.949086 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5a1c592-2f96-4918-847f-223fb5035a1c-console-oauth-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:29.949197 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:29.949094 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5a1c592-2f96-4918-847f-223fb5035a1c-oauth-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:44:30.447824 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:30.447756 2578 generic.go:356] "Generic (PLEG): container finished" podID="b5a1c592-2f96-4918-847f-223fb5035a1c" containerID="f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5" exitCode=0 Jul 08 01:44:30.447824 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:30.447818 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c5c7f74b-s79s8" Jul 08 01:44:30.448023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:30.447855 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c5c7f74b-s79s8" event={"ID":"b5a1c592-2f96-4918-847f-223fb5035a1c","Type":"ContainerDied","Data":"f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5"} Jul 08 01:44:30.448023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:30.447902 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c5c7f74b-s79s8" event={"ID":"b5a1c592-2f96-4918-847f-223fb5035a1c","Type":"ContainerDied","Data":"70443b2f9b385ebcb9a5db6d3fd379285e6d883d993a0db520500956d7d98eab"} Jul 08 01:44:30.448023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:30.447922 2578 scope.go:117] "RemoveContainer" containerID="f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5" Jul 08 01:44:30.456027 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:30.456007 2578 scope.go:117] "RemoveContainer" containerID="f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5" Jul 08 01:44:30.456259 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:44:30.456242 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5\": container with ID starting with f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5 not found: ID does not exist" containerID="f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5" Jul 08 01:44:30.456320 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:30.456265 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5"} err="failed to get container status \"f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5\": rpc error: code = NotFound desc = could not find container \"f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5\": container with ID starting with f3c5312f2c330aa4835079ddbab9396f7c18f7a9c0d3b5fd0c980a4209e1eeb5 not found: ID does not exist" Jul 08 01:44:30.464147 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:30.464123 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9c5c7f74b-s79s8"] Jul 08 01:44:30.467890 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:30.467871 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9c5c7f74b-s79s8"] Jul 08 01:44:31.904710 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:31.904672 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5a1c592-2f96-4918-847f-223fb5035a1c" path="/var/lib/kubelet/pods/b5a1c592-2f96-4918-847f-223fb5035a1c/volumes" Jul 08 01:44:32.850390 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:32.850359 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:44:32.854200 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:32.854181 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6845648df-vv42s" Jul 08 01:44:34.341761 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:34.341735 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-62q7j" Jul 08 01:44:54.687925 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:54.687884 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:44:54.707003 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:54.706979 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:44:55.561811 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:44:55.561787 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:12.733181 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:12.733147 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 01:45:12.733593 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:12.733416 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="prometheus" containerID="cri-o://4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" gracePeriod=600 Jul 08 01:45:12.733593 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:12.733433 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy" containerID="cri-o://0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" gracePeriod=600 Jul 08 01:45:12.733593 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:12.733451 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy-web" containerID="cri-o://1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" gracePeriod=600 Jul 08 01:45:12.733593 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:12.733509 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" gracePeriod=600 Jul 08 01:45:12.733593 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:12.733443 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="thanos-sidecar" containerID="cri-o://82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" gracePeriod=600 Jul 08 01:45:12.733593 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:12.733546 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="config-reloader" containerID="cri-o://ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" gracePeriod=600 Jul 08 01:45:12.962807 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:12.962784 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.091005 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.090983 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-1\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091134 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091017 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-grpc-tls\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091134 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091037 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-thanos-prometheus-http-client-file\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091199 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091159 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-web-config\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091199 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091191 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2ffm\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-kube-api-access-p2ffm\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091217 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-metrics-client-ca\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091241 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-tls\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091393 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091312 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:13.091393 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091366 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-kubelet-serving-ca-bundle\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091523 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091452 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-kube-rbac-proxy\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091523 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091508 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-0\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091638 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091547 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091638 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091572 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091638 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091596 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config-out\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091638 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091620 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-trusted-ca-bundle\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091646 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-serving-certs-ca-bundle\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091600 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:13.091842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091682 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091708 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-db\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091738 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-tls-assets\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.091842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091756 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:13.091842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.091770 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-2\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.092997 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.092971 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 01:45:13.094137 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.094112 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:13.094416 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.094393 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:13.094606 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.094529 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-kube-api-access-p2ffm" (OuterVolumeSpecName: "kube-api-access-p2ffm") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "kube-api-access-p2ffm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:45:13.094606 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.094572 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:13.094756 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.094674 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:13.094814 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.094778 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:13.095109 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.094882 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:13.095208 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095131 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:13.095208 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095179 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-metrics-client-certs\") pod \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\" (UID: \"6e45bebc-23b0-4b1b-9bb1-2964d441c271\") " Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095442 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095466 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095510 2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-grpc-tls\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095526 2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095542 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2ffm\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-kube-api-access-p2ffm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095556 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-metrics-client-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095570 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095584 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095599 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095615 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095630 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095645 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-db\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095175 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:45:13.095659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095583 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:13.096431 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.095736 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:13.096569 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.096544 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config" (OuterVolumeSpecName: "config") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:13.096747 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.096723 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:13.097059 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.097041 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config-out" (OuterVolumeSpecName: "config-out") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 01:45:13.097180 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.097166 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:13.104759 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.104739 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-web-config" (OuterVolumeSpecName: "web-config") pod "6e45bebc-23b0-4b1b-9bb1-2964d441c271" (UID: "6e45bebc-23b0-4b1b-9bb1-2964d441c271"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:13.200694 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.197357 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-web-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.200694 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.197385 2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-kube-rbac-proxy\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.200694 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.197401 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.200694 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.197421 2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.200694 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.197435 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e45bebc-23b0-4b1b-9bb1-2964d441c271-config-out\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.200694 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.197449 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e45bebc-23b0-4b1b-9bb1-2964d441c271-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.200694 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.197462 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e45bebc-23b0-4b1b-9bb1-2964d441c271-tls-assets\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.200694 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.197497 2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e45bebc-23b0-4b1b-9bb1-2964d441c271-secret-metrics-client-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:13.611513 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611473 2578 generic.go:356] "Generic (PLEG): container finished" podID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerID="cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" exitCode=0 Jul 08 01:45:13.611513 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611508 2578 generic.go:356] "Generic (PLEG): container finished" podID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerID="0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" exitCode=0 Jul 08 01:45:13.611513 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611514 2578 generic.go:356] "Generic (PLEG): container finished" podID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerID="1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" exitCode=0 Jul 08 01:45:13.611513 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611519 2578 generic.go:356] "Generic (PLEG): container finished" podID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerID="82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" exitCode=0 Jul 08 01:45:13.611748 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611525 2578 generic.go:356] "Generic (PLEG): container finished" podID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerID="ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" exitCode=0 Jul 08 01:45:13.611748 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611531 2578 generic.go:356] "Generic (PLEG): container finished" podID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerID="4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" exitCode=0 Jul 08 01:45:13.611748 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611562 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerDied","Data":"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341"} Jul 08 01:45:13.611748 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611572 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.611748 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611643 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerDied","Data":"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e"} Jul 08 01:45:13.611748 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611670 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerDied","Data":"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067"} Jul 08 01:45:13.611748 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611693 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerDied","Data":"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829"} Jul 08 01:45:13.611748 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611717 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerDied","Data":"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1"} Jul 08 01:45:13.611748 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611738 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerDied","Data":"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b"} Jul 08 01:45:13.612104 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611724 2578 scope.go:117] "RemoveContainer" containerID="cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" Jul 08 01:45:13.612104 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.611761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e45bebc-23b0-4b1b-9bb1-2964d441c271","Type":"ContainerDied","Data":"d742f5e1c9cbf2e6351031b9a8069cc1c6c6adcbdeb003ccbf210c2b806080b9"} Jul 08 01:45:13.619354 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.619327 2578 scope.go:117] "RemoveContainer" containerID="0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" Jul 08 01:45:13.626346 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.626330 2578 scope.go:117] "RemoveContainer" containerID="1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" Jul 08 01:45:13.632532 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.632513 2578 scope.go:117] "RemoveContainer" containerID="82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" Jul 08 01:45:13.636194 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.636171 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 01:45:13.639091 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.639075 2578 scope.go:117] "RemoveContainer" containerID="ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" Jul 08 01:45:13.642260 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.642241 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 01:45:13.645591 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.645576 2578 scope.go:117] "RemoveContainer" containerID="4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" Jul 08 01:45:13.652132 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.652118 2578 scope.go:117] "RemoveContainer" containerID="f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b" Jul 08 01:45:13.658093 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.658077 2578 scope.go:117] "RemoveContainer" containerID="cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" Jul 08 01:45:13.658423 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:45:13.658396 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": container with ID starting with cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341 not found: ID does not exist" containerID="cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" Jul 08 01:45:13.658549 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.658426 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341"} err="failed to get container status \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": rpc error: code = NotFound desc = could not find container \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": container with ID starting with cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341 not found: ID does not exist" Jul 08 01:45:13.658549 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.658449 2578 scope.go:117] "RemoveContainer" containerID="0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" Jul 08 01:45:13.658842 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:45:13.658736 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": container with ID starting with 0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e not found: ID does not exist" containerID="0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" Jul 08 01:45:13.658842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.658765 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e"} err="failed to get container status \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": rpc error: code = NotFound desc = could not find container \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": container with ID starting with 0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e not found: ID does not exist" Jul 08 01:45:13.658842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.658784 2578 scope.go:117] "RemoveContainer" containerID="1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" Jul 08 01:45:13.659079 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:45:13.659060 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": container with ID starting with 1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067 not found: ID does not exist" containerID="1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" Jul 08 01:45:13.659119 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.659086 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067"} err="failed to get container status \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": rpc error: code = NotFound desc = could not find container \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": container with ID starting with 1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067 not found: ID does not exist" Jul 08 01:45:13.659119 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.659102 2578 scope.go:117] "RemoveContainer" containerID="82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" Jul 08 01:45:13.659334 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:45:13.659316 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": container with ID starting with 82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829 not found: ID does not exist" containerID="82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" Jul 08 01:45:13.659369 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.659340 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829"} err="failed to get container status \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": rpc error: code = NotFound desc = could not find container \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": container with ID starting with 82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829 not found: ID does not exist" Jul 08 01:45:13.659369 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.659356 2578 scope.go:117] "RemoveContainer" containerID="ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" Jul 08 01:45:13.659606 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:45:13.659589 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": container with ID starting with ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1 not found: ID does not exist" containerID="ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" Jul 08 01:45:13.659656 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.659611 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1"} err="failed to get container status \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": rpc error: code = NotFound desc = could not find container \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": container with ID starting with ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1 not found: ID does not exist" Jul 08 01:45:13.659656 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.659633 2578 scope.go:117] "RemoveContainer" containerID="4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" Jul 08 01:45:13.659870 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:45:13.659836 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": container with ID starting with 4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b not found: ID does not exist" containerID="4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" Jul 08 01:45:13.659909 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.659874 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b"} err="failed to get container status \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": rpc error: code = NotFound desc = could not find container \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": container with ID starting with 4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b not found: ID does not exist" Jul 08 01:45:13.659909 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.659887 2578 scope.go:117] "RemoveContainer" containerID="f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b" Jul 08 01:45:13.660102 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:45:13.660087 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": container with ID starting with f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b not found: ID does not exist" containerID="f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b" Jul 08 01:45:13.660155 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660105 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b"} err="failed to get container status \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": rpc error: code = NotFound desc = could not find container \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": container with ID starting with f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b not found: ID does not exist" Jul 08 01:45:13.660155 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660116 2578 scope.go:117] "RemoveContainer" containerID="cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" Jul 08 01:45:13.660333 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660317 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341"} err="failed to get container status \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": rpc error: code = NotFound desc = could not find container \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": container with ID starting with cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341 not found: ID does not exist" Jul 08 01:45:13.660372 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660334 2578 scope.go:117] "RemoveContainer" containerID="0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" Jul 08 01:45:13.660548 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660529 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e"} err="failed to get container status \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": rpc error: code = NotFound desc = could not find container \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": container with ID starting with 0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e not found: ID does not exist" Jul 08 01:45:13.660595 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660549 2578 scope.go:117] "RemoveContainer" containerID="1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" Jul 08 01:45:13.660760 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660743 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067"} err="failed to get container status \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": rpc error: code = NotFound desc = could not find container \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": container with ID starting with 1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067 not found: ID does not exist" Jul 08 01:45:13.660804 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660760 2578 scope.go:117] "RemoveContainer" containerID="82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" Jul 08 01:45:13.660955 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660941 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829"} err="failed to get container status \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": rpc error: code = NotFound desc = could not find container \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": container with ID starting with 82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829 not found: ID does not exist" Jul 08 01:45:13.660995 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.660954 2578 scope.go:117] "RemoveContainer" containerID="ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" Jul 08 01:45:13.661150 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661131 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1"} err="failed to get container status \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": rpc error: code = NotFound desc = could not find container \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": container with ID starting with ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1 not found: ID does not exist" Jul 08 01:45:13.661195 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661150 2578 scope.go:117] "RemoveContainer" containerID="4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" Jul 08 01:45:13.661342 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661325 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b"} err="failed to get container status \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": rpc error: code = NotFound desc = could not find container \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": container with ID starting with 4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b not found: ID does not exist" Jul 08 01:45:13.661376 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661343 2578 scope.go:117] "RemoveContainer" containerID="f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b" Jul 08 01:45:13.661563 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661544 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b"} err="failed to get container status \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": rpc error: code = NotFound desc = could not find container \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": container with ID starting with f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b not found: ID does not exist" Jul 08 01:45:13.661638 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661564 2578 scope.go:117] "RemoveContainer" containerID="cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" Jul 08 01:45:13.661753 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661737 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341"} err="failed to get container status \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": rpc error: code = NotFound desc = could not find container \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": container with ID starting with cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341 not found: ID does not exist" Jul 08 01:45:13.661799 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661753 2578 scope.go:117] "RemoveContainer" containerID="0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" Jul 08 01:45:13.661920 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661904 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e"} err="failed to get container status \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": rpc error: code = NotFound desc = could not find container \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": container with ID starting with 0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e not found: ID does not exist" Jul 08 01:45:13.661959 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.661920 2578 scope.go:117] "RemoveContainer" containerID="1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" Jul 08 01:45:13.662101 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662082 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067"} err="failed to get container status \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": rpc error: code = NotFound desc = could not find container \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": container with ID starting with 1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067 not found: ID does not exist" Jul 08 01:45:13.662141 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662101 2578 scope.go:117] "RemoveContainer" containerID="82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" Jul 08 01:45:13.662293 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662277 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829"} err="failed to get container status \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": rpc error: code = NotFound desc = could not find container \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": container with ID starting with 82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829 not found: ID does not exist" Jul 08 01:45:13.662336 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662292 2578 scope.go:117] "RemoveContainer" containerID="ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" Jul 08 01:45:13.662465 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662450 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1"} err="failed to get container status \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": rpc error: code = NotFound desc = could not find container \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": container with ID starting with ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1 not found: ID does not exist" Jul 08 01:45:13.662534 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662465 2578 scope.go:117] "RemoveContainer" containerID="4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" Jul 08 01:45:13.662636 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662620 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b"} err="failed to get container status \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": rpc error: code = NotFound desc = could not find container \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": container with ID starting with 4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b not found: ID does not exist" Jul 08 01:45:13.662677 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662637 2578 scope.go:117] "RemoveContainer" containerID="f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b" Jul 08 01:45:13.662811 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662796 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b"} err="failed to get container status \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": rpc error: code = NotFound desc = could not find container \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": container with ID starting with f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b not found: ID does not exist" Jul 08 01:45:13.662860 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662813 2578 scope.go:117] "RemoveContainer" containerID="cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" Jul 08 01:45:13.662978 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662961 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341"} err="failed to get container status \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": rpc error: code = NotFound desc = could not find container \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": container with ID starting with cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341 not found: ID does not exist" Jul 08 01:45:13.663016 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.662978 2578 scope.go:117] "RemoveContainer" containerID="0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" Jul 08 01:45:13.663161 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.663143 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e"} err="failed to get container status \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": rpc error: code = NotFound desc = could not find container \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": container with ID starting with 0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e not found: ID does not exist" Jul 08 01:45:13.663200 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.663163 2578 scope.go:117] "RemoveContainer" containerID="1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" Jul 08 01:45:13.663362 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.663345 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067"} err="failed to get container status \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": rpc error: code = NotFound desc = could not find container \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": container with ID starting with 1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067 not found: ID does not exist" Jul 08 01:45:13.663408 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.663362 2578 scope.go:117] "RemoveContainer" containerID="82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" Jul 08 01:45:13.663591 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.663575 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829"} err="failed to get container status \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": rpc error: code = NotFound desc = could not find container \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": container with ID starting with 82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829 not found: ID does not exist" Jul 08 01:45:13.663636 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.663592 2578 scope.go:117] "RemoveContainer" containerID="ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" Jul 08 01:45:13.663799 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.663783 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1"} err="failed to get container status \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": rpc error: code = NotFound desc = could not find container \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": container with ID starting with ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1 not found: ID does not exist" Jul 08 01:45:13.663863 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.663800 2578 scope.go:117] "RemoveContainer" containerID="4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" Jul 08 01:45:13.664012 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.663995 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b"} err="failed to get container status \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": rpc error: code = NotFound desc = could not find container \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": container with ID starting with 4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b not found: ID does not exist" Jul 08 01:45:13.664051 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.664012 2578 scope.go:117] "RemoveContainer" containerID="f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b" Jul 08 01:45:13.664219 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.664203 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b"} err="failed to get container status \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": rpc error: code = NotFound desc = could not find container \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": container with ID starting with f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b not found: ID does not exist" Jul 08 01:45:13.664260 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.664218 2578 scope.go:117] "RemoveContainer" containerID="cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" Jul 08 01:45:13.664424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.664406 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341"} err="failed to get container status \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": rpc error: code = NotFound desc = could not find container \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": container with ID starting with cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341 not found: ID does not exist" Jul 08 01:45:13.664463 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.664425 2578 scope.go:117] "RemoveContainer" containerID="0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" Jul 08 01:45:13.664643 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.664625 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e"} err="failed to get container status \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": rpc error: code = NotFound desc = could not find container \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": container with ID starting with 0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e not found: ID does not exist" Jul 08 01:45:13.664643 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.664642 2578 scope.go:117] "RemoveContainer" containerID="1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" Jul 08 01:45:13.664849 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.664827 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067"} err="failed to get container status \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": rpc error: code = NotFound desc = could not find container \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": container with ID starting with 1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067 not found: ID does not exist" Jul 08 01:45:13.664895 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.664850 2578 scope.go:117] "RemoveContainer" containerID="82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" Jul 08 01:45:13.665050 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665035 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829"} err="failed to get container status \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": rpc error: code = NotFound desc = could not find container \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": container with ID starting with 82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829 not found: ID does not exist" Jul 08 01:45:13.665084 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665049 2578 scope.go:117] "RemoveContainer" containerID="ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" Jul 08 01:45:13.665228 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665211 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1"} err="failed to get container status \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": rpc error: code = NotFound desc = could not find container \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": container with ID starting with ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1 not found: ID does not exist" Jul 08 01:45:13.665269 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665228 2578 scope.go:117] "RemoveContainer" containerID="4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" Jul 08 01:45:13.665433 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665413 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b"} err="failed to get container status \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": rpc error: code = NotFound desc = could not find container \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": container with ID starting with 4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b not found: ID does not exist" Jul 08 01:45:13.665491 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665434 2578 scope.go:117] "RemoveContainer" containerID="f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b" Jul 08 01:45:13.665678 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665660 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b"} err="failed to get container status \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": rpc error: code = NotFound desc = could not find container \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": container with ID starting with f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b not found: ID does not exist" Jul 08 01:45:13.665715 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665678 2578 scope.go:117] "RemoveContainer" containerID="cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341" Jul 08 01:45:13.665878 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665859 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341"} err="failed to get container status \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": rpc error: code = NotFound desc = could not find container \"cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341\": container with ID starting with cbb39f61637e1a40c6151945d14ca09d6fb2b279082efba5e26dda4e992e7341 not found: ID does not exist" Jul 08 01:45:13.665917 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.665879 2578 scope.go:117] "RemoveContainer" containerID="0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e" Jul 08 01:45:13.666338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.666076 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e"} err="failed to get container status \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": rpc error: code = NotFound desc = could not find container \"0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e\": container with ID starting with 0ccb7396fd93d618c6ab190a627a088757133e8f9f7e732654df8be506ccba6e not found: ID does not exist" Jul 08 01:45:13.666338 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.666133 2578 scope.go:117] "RemoveContainer" containerID="1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067" Jul 08 01:45:13.666469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.666372 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067"} err="failed to get container status \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": rpc error: code = NotFound desc = could not find container \"1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067\": container with ID starting with 1455330b03f5c747f153290b6e93b85cc5332db1db711c023ce90e4af6974067 not found: ID does not exist" Jul 08 01:45:13.666469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.666398 2578 scope.go:117] "RemoveContainer" containerID="82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829" Jul 08 01:45:13.666700 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.666675 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829"} err="failed to get container status \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": rpc error: code = NotFound desc = could not find container \"82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829\": container with ID starting with 82b73cae1933ada9a9a2cf4a6a9d1aebd4d787827e739430016f74ff014b0829 not found: ID does not exist" Jul 08 01:45:13.666780 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.666703 2578 scope.go:117] "RemoveContainer" containerID="ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1" Jul 08 01:45:13.667033 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.666965 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1"} err="failed to get container status \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": rpc error: code = NotFound desc = could not find container \"ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1\": container with ID starting with ab98969fece86253dd85acfa461b22311129358ddb742c2fa7d26626530a1fc1 not found: ID does not exist" Jul 08 01:45:13.667033 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.666993 2578 scope.go:117] "RemoveContainer" containerID="4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b" Jul 08 01:45:13.667243 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.667221 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b"} err="failed to get container status \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": rpc error: code = NotFound desc = could not find container \"4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b\": container with ID starting with 4ace801b74cab832e5a32c8d4a075f0c26c7a88cd0d87b8a3ec0585d6d672c8b not found: ID does not exist" Jul 08 01:45:13.667300 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.667245 2578 scope.go:117] "RemoveContainer" containerID="f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b" Jul 08 01:45:13.667511 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.667491 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b"} err="failed to get container status \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": rpc error: code = NotFound desc = could not find container \"f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b\": container with ID starting with f0268159588cfecb7ed6e4ba4e1e6da18a5e7af38a031cf97cbd4fbd7f49658b not found: ID does not exist" Jul 08 01:45:13.667904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.667885 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 01:45:13.668159 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668145 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a1c592-2f96-4918-847f-223fb5035a1c" containerName="console" Jul 08 01:45:13.668222 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668162 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a1c592-2f96-4918-847f-223fb5035a1c" containerName="console" Jul 08 01:45:13.668222 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668177 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="627218e3-b587-482b-b1c3-f3a9ebe5b0d4" containerName="console" Jul 08 01:45:13.668222 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668186 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="627218e3-b587-482b-b1c3-f3a9ebe5b0d4" containerName="console" Jul 08 01:45:13.668222 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668199 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="config-reloader" Jul 08 01:45:13.668222 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668208 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="config-reloader" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668224 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="init-config-reloader" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668233 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="init-config-reloader" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668244 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="thanos-sidecar" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668253 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="thanos-sidecar" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668264 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy-web" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668273 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy-web" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668290 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="prometheus" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668299 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="prometheus" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668310 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668319 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668328 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy-thanos" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668337 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy-thanos" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668408 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a1c592-2f96-4918-847f-223fb5035a1c" containerName="console" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668419 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="config-reloader" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668428 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="thanos-sidecar" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668437 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy-web" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668446 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668455 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="627218e3-b587-482b-b1c3-f3a9ebe5b0d4" containerName="console" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668465 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="prometheus" Jul 08 01:45:13.668469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.668489 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" containerName="kube-rbac-proxy-thanos" Jul 08 01:45:13.672197 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.672179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.675096 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675076 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-uq3tj0ovietm\"" Jul 08 01:45:13.675096 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675087 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 01:45:13.675244 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675077 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 01:45:13.675244 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675128 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 01:45:13.675244 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675077 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 01:45:13.675244 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675191 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 01:45:13.675244 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675077 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 01:45:13.675489 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675295 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 01:45:13.675586 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675571 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 01:45:13.675683 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675665 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 01:45:13.675683 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.675679 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 01:45:13.676220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.676198 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 01:45:13.676220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.676213 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xvxlq\"" Jul 08 01:45:13.676220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.676232 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 01:45:13.678220 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.678201 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 01:45:13.680872 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.680854 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 01:45:13.692528 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.689543 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 01:45:13.801730 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.801709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.801741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrbd\" (UniqueName: \"kubernetes.io/projected/93f0548d-c223-4f5a-8656-691aa551e5a1-kube-api-access-vsrbd\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.801761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-web-config\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.801803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.801832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.801864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.801890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.801907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802023 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.801985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93f0548d-c223-4f5a-8656-691aa551e5a1-config-out\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802057 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-config\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93f0548d-c223-4f5a-8656-691aa551e5a1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.802290 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.802225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902613 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93f0548d-c223-4f5a-8656-691aa551e5a1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902613 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902613 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrbd\" (UniqueName: \"kubernetes.io/projected/93f0548d-c223-4f5a-8656-691aa551e5a1-kube-api-access-vsrbd\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-web-config\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902732 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.902845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93f0548d-c223-4f5a-8656-691aa551e5a1-config-out\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.902999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.903036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.903068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-config\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903639 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.903379 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.903639 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.903425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.905257 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.904366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.905257 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.904588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.905495 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.905316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.905687 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.905658 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.906205 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.906179 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.906494 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.906443 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93f0548d-c223-4f5a-8656-691aa551e5a1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.906648 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.906624 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.907938 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.907917 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.908415 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.908327 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.908415 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.908335 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e45bebc-23b0-4b1b-9bb1-2964d441c271" path="/var/lib/kubelet/pods/6e45bebc-23b0-4b1b-9bb1-2964d441c271/volumes" Jul 08 01:45:13.908415 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.908364 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-web-config\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.908415 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.908386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93f0548d-c223-4f5a-8656-691aa551e5a1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.908648 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.908627 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.908842 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.908804 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.909000 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.908982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.909062 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.908996 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-config\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.909170 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.909152 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93f0548d-c223-4f5a-8656-691aa551e5a1-config-out\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.909695 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.909678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93f0548d-c223-4f5a-8656-691aa551e5a1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.911567 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.911546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrbd\" (UniqueName: \"kubernetes.io/projected/93f0548d-c223-4f5a-8656-691aa551e5a1-kube-api-access-vsrbd\") pod \"prometheus-k8s-0\" (UID: \"93f0548d-c223-4f5a-8656-691aa551e5a1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:13.982094 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:13.982069 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:14.120186 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:14.120166 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 01:45:14.122098 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:45:14.122069 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f0548d_c223_4f5a_8656_691aa551e5a1.slice/crio-427b2fc5fe433200b70262ed83b036477ba3fcee74e6aacaa0f500c9ef94c438 WatchSource:0}: Error finding container 427b2fc5fe433200b70262ed83b036477ba3fcee74e6aacaa0f500c9ef94c438: Status 404 returned error can't find the container with id 427b2fc5fe433200b70262ed83b036477ba3fcee74e6aacaa0f500c9ef94c438 Jul 08 01:45:14.616787 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:14.616755 2578 generic.go:356] "Generic (PLEG): container finished" podID="93f0548d-c223-4f5a-8656-691aa551e5a1" containerID="eeee8e0f1e66b7b96afb209d75ef51a60e09deddeb6eedd7c11fbf4a759dca4b" exitCode=0 Jul 08 01:45:14.616955 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:14.616840 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93f0548d-c223-4f5a-8656-691aa551e5a1","Type":"ContainerDied","Data":"eeee8e0f1e66b7b96afb209d75ef51a60e09deddeb6eedd7c11fbf4a759dca4b"} Jul 08 01:45:14.616955 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:14.616909 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93f0548d-c223-4f5a-8656-691aa551e5a1","Type":"ContainerStarted","Data":"427b2fc5fe433200b70262ed83b036477ba3fcee74e6aacaa0f500c9ef94c438"} Jul 08 01:45:15.626952 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:15.626912 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93f0548d-c223-4f5a-8656-691aa551e5a1","Type":"ContainerStarted","Data":"120ae497e9a1c880dc7c6be99038e108003ba82802aa4603cb66b2b04d00a893"} Jul 08 01:45:15.627436 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:15.627024 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93f0548d-c223-4f5a-8656-691aa551e5a1","Type":"ContainerStarted","Data":"3f50887e1ec851b9e0e2bc5dc58ccd4fc7642bc0fa366bfd2107cb5271ccd32f"} Jul 08 01:45:15.627436 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:15.627071 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93f0548d-c223-4f5a-8656-691aa551e5a1","Type":"ContainerStarted","Data":"c2d5c1087487cc1ea6743e73710d839f0ccc230c05c3fc1788e4d73b22f5e2b4"} Jul 08 01:45:15.627436 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:15.627116 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93f0548d-c223-4f5a-8656-691aa551e5a1","Type":"ContainerStarted","Data":"e4259af25297ac404f4798eb678bdab4ca5928d37842e8ea7d224d72ecfb41a7"} Jul 08 01:45:15.627436 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:15.627155 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93f0548d-c223-4f5a-8656-691aa551e5a1","Type":"ContainerStarted","Data":"92f2d079ae84b1b471e27b7f6a423b210351b45eca26e8fc8e753310b95494b5"} Jul 08 01:45:15.627436 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:15.627195 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93f0548d-c223-4f5a-8656-691aa551e5a1","Type":"ContainerStarted","Data":"17129d9725286b6d8b295a7c9f0f58d48d346de645b4c254a1e55b7889b2b7fc"} Jul 08 01:45:15.657115 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:15.657065 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.638434641 podStartE2EDuration="2.657049065s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.614424ms" isStatefulPod=false podCreationTimestamp="2026-07-08 01:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:45:15.654433093 +0000 UTC m=+146.316347629" watchObservedRunningTime="2026-07-08 01:45:15.657049065 +0000 UTC m=+146.318963602" Jul 08 01:45:16.484323 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.484293 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c88fcd5c-55mp6"] Jul 08 01:45:16.486444 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.486427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.499113 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.499089 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c88fcd5c-55mp6"] Jul 08 01:45:16.525741 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.525714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-oauth-config\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.525857 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.525745 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-service-ca\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.525857 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.525768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-serving-cert\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.525857 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.525790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-config\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.525857 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.525835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-trusted-ca-bundle\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.526006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.525869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-oauth-serving-cert\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.526006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.525904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4fc\" (UniqueName: \"kubernetes.io/projected/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-kube-api-access-kh4fc\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.627095 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.627063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-oauth-config\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.627524 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.627105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-service-ca\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.627524 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.627131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-serving-cert\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.627524 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.627154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-config\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.627524 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.627256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-trusted-ca-bundle\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.627524 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.627298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-oauth-serving-cert\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.627524 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.627320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4fc\" (UniqueName: \"kubernetes.io/projected/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-kube-api-access-kh4fc\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.627954 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.627932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-config\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.628177 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.628090 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-oauth-serving-cert\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.628177 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.628100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-service-ca\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.628289 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.628177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-trusted-ca-bundle\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.629904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.629884 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-serving-cert\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.630105 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.630083 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-oauth-config\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.634978 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.634957 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4fc\" (UniqueName: \"kubernetes.io/projected/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-kube-api-access-kh4fc\") pod \"console-6c88fcd5c-55mp6\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.795368 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.795301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:16.917569 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:16.917435 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c88fcd5c-55mp6"] Jul 08 01:45:16.920184 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:45:16.920156 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf4c8a6_5e6b_4640_96b9_f09bfda0e994.slice/crio-2f433d50d7fc7804b1a402e138122a0602818c69bcc408c4e1d0785be44c107b WatchSource:0}: Error finding container 2f433d50d7fc7804b1a402e138122a0602818c69bcc408c4e1d0785be44c107b: Status 404 returned error can't find the container with id 2f433d50d7fc7804b1a402e138122a0602818c69bcc408c4e1d0785be44c107b Jul 08 01:45:17.635774 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:17.635738 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c88fcd5c-55mp6" event={"ID":"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994","Type":"ContainerStarted","Data":"910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20"} Jul 08 01:45:17.636168 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:17.635828 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c88fcd5c-55mp6" event={"ID":"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994","Type":"ContainerStarted","Data":"2f433d50d7fc7804b1a402e138122a0602818c69bcc408c4e1d0785be44c107b"} Jul 08 01:45:17.656394 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:17.656352 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c88fcd5c-55mp6" podStartSLOduration=1.656336955 podStartE2EDuration="1.656336955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:45:17.654648664 +0000 UTC m=+148.316563201" watchObservedRunningTime="2026-07-08 01:45:17.656336955 +0000 UTC m=+148.318251492" Jul 08 01:45:18.982344 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:18.982306 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:45:26.796015 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:26.795978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:26.796520 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:26.796059 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:26.800806 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:26.800789 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:27.676659 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:27.676631 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:45:27.720619 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:27.720592 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-668457bc49-zfp8t"] Jul 08 01:45:52.740625 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:52.740560 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-668457bc49-zfp8t" podUID="679a371f-a778-476a-94b8-2ae02a3d5605" containerName="console" containerID="cri-o://3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6" gracePeriod=15 Jul 08 01:45:52.986900 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:52.986878 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:45:53.111773 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.111749 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-trusted-ca-bundle\") pod \"679a371f-a778-476a-94b8-2ae02a3d5605\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " Jul 08 01:45:53.111912 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.111782 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-service-ca\") pod \"679a371f-a778-476a-94b8-2ae02a3d5605\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " Jul 08 01:45:53.111912 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.111797 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-oauth-serving-cert\") pod \"679a371f-a778-476a-94b8-2ae02a3d5605\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " Jul 08 01:45:53.111912 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.111835 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46n7j\" (UniqueName: \"kubernetes.io/projected/679a371f-a778-476a-94b8-2ae02a3d5605-kube-api-access-46n7j\") pod \"679a371f-a778-476a-94b8-2ae02a3d5605\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " Jul 08 01:45:53.112030 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.111944 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-console-config\") pod \"679a371f-a778-476a-94b8-2ae02a3d5605\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " Jul 08 01:45:53.112030 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.111982 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-serving-cert\") pod \"679a371f-a778-476a-94b8-2ae02a3d5605\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " Jul 08 01:45:53.112030 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.112021 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-oauth-config\") pod \"679a371f-a778-476a-94b8-2ae02a3d5605\" (UID: \"679a371f-a778-476a-94b8-2ae02a3d5605\") " Jul 08 01:45:53.112269 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.112232 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-service-ca" (OuterVolumeSpecName: "service-ca") pod "679a371f-a778-476a-94b8-2ae02a3d5605" (UID: "679a371f-a778-476a-94b8-2ae02a3d5605"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:53.112269 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.112244 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "679a371f-a778-476a-94b8-2ae02a3d5605" (UID: "679a371f-a778-476a-94b8-2ae02a3d5605"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:53.112269 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.112247 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "679a371f-a778-476a-94b8-2ae02a3d5605" (UID: "679a371f-a778-476a-94b8-2ae02a3d5605"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:53.112451 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.112289 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-console-config" (OuterVolumeSpecName: "console-config") pod "679a371f-a778-476a-94b8-2ae02a3d5605" (UID: "679a371f-a778-476a-94b8-2ae02a3d5605"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:45:53.114034 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.114006 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679a371f-a778-476a-94b8-2ae02a3d5605-kube-api-access-46n7j" (OuterVolumeSpecName: "kube-api-access-46n7j") pod "679a371f-a778-476a-94b8-2ae02a3d5605" (UID: "679a371f-a778-476a-94b8-2ae02a3d5605"). InnerVolumeSpecName "kube-api-access-46n7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:45:53.114146 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.114049 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "679a371f-a778-476a-94b8-2ae02a3d5605" (UID: "679a371f-a778-476a-94b8-2ae02a3d5605"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:53.114207 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.114163 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "679a371f-a778-476a-94b8-2ae02a3d5605" (UID: "679a371f-a778-476a-94b8-2ae02a3d5605"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:45:53.213520 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.213493 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-oauth-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:53.213520 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.213516 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-trusted-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:53.213520 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.213526 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-service-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:53.213713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.213534 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-oauth-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:53.213713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.213543 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46n7j\" (UniqueName: \"kubernetes.io/projected/679a371f-a778-476a-94b8-2ae02a3d5605-kube-api-access-46n7j\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:53.213713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.213551 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/679a371f-a778-476a-94b8-2ae02a3d5605-console-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:53.213713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.213562 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/679a371f-a778-476a-94b8-2ae02a3d5605-console-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:45:53.772812 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.772777 2578 generic.go:356] "Generic (PLEG): container finished" podID="679a371f-a778-476a-94b8-2ae02a3d5605" containerID="3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6" exitCode=0 Jul 08 01:45:53.773235 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.772826 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668457bc49-zfp8t" Jul 08 01:45:53.773235 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.772849 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668457bc49-zfp8t" event={"ID":"679a371f-a778-476a-94b8-2ae02a3d5605","Type":"ContainerDied","Data":"3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6"} Jul 08 01:45:53.773235 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.772903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668457bc49-zfp8t" event={"ID":"679a371f-a778-476a-94b8-2ae02a3d5605","Type":"ContainerDied","Data":"91d3aec1c3ab2a4a85fb6cff8da17c295fdfb7fddbb48279e91f72a2f7cfc9e4"} Jul 08 01:45:53.773235 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.772930 2578 scope.go:117] "RemoveContainer" containerID="3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6" Jul 08 01:45:53.781309 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.781294 2578 scope.go:117] "RemoveContainer" containerID="3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6" Jul 08 01:45:53.781572 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:45:53.781547 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6\": container with ID starting with 3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6 not found: ID does not exist" containerID="3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6" Jul 08 01:45:53.781697 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.781581 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6"} err="failed to get container status \"3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6\": rpc error: code = NotFound desc = could not find container \"3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6\": container with ID starting with 3b148e7c857afc8605adcb2b3f0bce1b69fe4b69afab9a6601a5358548887ba6 not found: ID does not exist" Jul 08 01:45:53.792533 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.792511 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-668457bc49-zfp8t"] Jul 08 01:45:53.796097 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.796079 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-668457bc49-zfp8t"] Jul 08 01:45:53.905012 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:45:53.904986 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="679a371f-a778-476a-94b8-2ae02a3d5605" path="/var/lib/kubelet/pods/679a371f-a778-476a-94b8-2ae02a3d5605/volumes" Jul 08 01:46:13.983140 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:13.983105 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:46:13.998189 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:13.998163 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:46:14.863078 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:14.863050 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 01:46:54.919590 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:54.919557 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f88f7fc9d-tkzln"] Jul 08 01:46:54.920021 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:54.919901 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679a371f-a778-476a-94b8-2ae02a3d5605" containerName="console" Jul 08 01:46:54.920021 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:54.919919 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="679a371f-a778-476a-94b8-2ae02a3d5605" containerName="console" Jul 08 01:46:54.920021 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:54.919982 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="679a371f-a778-476a-94b8-2ae02a3d5605" containerName="console" Jul 08 01:46:54.922587 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:54.922571 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:54.939110 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:54.939084 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f88f7fc9d-tkzln"] Jul 08 01:46:55.005388 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.005358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-trusted-ca-bundle\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.005388 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.005388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-config\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.005615 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.005407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-service-ca\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.005615 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.005424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9zs\" (UniqueName: \"kubernetes.io/projected/64fc3648-ced9-44e4-b3a8-612561c26ebb-kube-api-access-2w9zs\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.005615 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.005514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-oauth-config\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.005615 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.005560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-serving-cert\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.005775 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.005650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-oauth-serving-cert\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.106252 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.106213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-oauth-serving-cert\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.106539 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.106262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-trusted-ca-bundle\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.106539 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.106286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-config\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.106539 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.106306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-service-ca\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.106539 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.106328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9zs\" (UniqueName: \"kubernetes.io/projected/64fc3648-ced9-44e4-b3a8-612561c26ebb-kube-api-access-2w9zs\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.106539 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.106350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-oauth-config\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.106539 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.106384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-serving-cert\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.107066 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.107032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-config\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.107149 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.107037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-oauth-serving-cert\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.107207 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.107181 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-service-ca\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.107257 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.107202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-trusted-ca-bundle\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.108841 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.108818 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-oauth-config\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.108998 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.108982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-serving-cert\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.113966 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.113932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9zs\" (UniqueName: \"kubernetes.io/projected/64fc3648-ced9-44e4-b3a8-612561c26ebb-kube-api-access-2w9zs\") pod \"console-f88f7fc9d-tkzln\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.231288 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.231197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:46:55.354757 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:55.354722 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f88f7fc9d-tkzln"] Jul 08 01:46:55.357805 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:46:55.357777 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fc3648_ced9_44e4_b3a8_612561c26ebb.slice/crio-13138f91c6c32ef7a9774422aa3b72b6a9d8f14bd3d5cab32d37185c656c362f WatchSource:0}: Error finding container 13138f91c6c32ef7a9774422aa3b72b6a9d8f14bd3d5cab32d37185c656c362f: Status 404 returned error can't find the container with id 13138f91c6c32ef7a9774422aa3b72b6a9d8f14bd3d5cab32d37185c656c362f Jul 08 01:46:56.000153 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:56.000120 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f88f7fc9d-tkzln" event={"ID":"64fc3648-ced9-44e4-b3a8-612561c26ebb","Type":"ContainerStarted","Data":"916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed"} Jul 08 01:46:56.000531 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:56.000172 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f88f7fc9d-tkzln" event={"ID":"64fc3648-ced9-44e4-b3a8-612561c26ebb","Type":"ContainerStarted","Data":"13138f91c6c32ef7a9774422aa3b72b6a9d8f14bd3d5cab32d37185c656c362f"} Jul 08 01:46:56.017266 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:46:56.017190 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f88f7fc9d-tkzln" podStartSLOduration=2.017175241 podStartE2EDuration="2.017175241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:46:56.016615162 +0000 UTC m=+246.678529703" watchObservedRunningTime="2026-07-08 01:46:56.017175241 +0000 UTC m=+246.679089778" Jul 08 01:47:05.231413 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:05.231372 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:47:05.231855 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:05.231445 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:47:05.236224 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:05.236198 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:47:06.039311 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:06.039275 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:47:06.084890 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:06.084853 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c88fcd5c-55mp6"] Jul 08 01:47:31.106110 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.106046 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6c88fcd5c-55mp6" podUID="ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" containerName="console" containerID="cri-o://910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20" gracePeriod=15 Jul 08 01:47:31.342263 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.342239 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:47:31.388834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.388754 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-oauth-config\") pod \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " Jul 08 01:47:31.388834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.388791 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh4fc\" (UniqueName: \"kubernetes.io/projected/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-kube-api-access-kh4fc\") pod \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " Jul 08 01:47:31.388834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.388814 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-oauth-serving-cert\") pod \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " Jul 08 01:47:31.388834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.388832 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-serving-cert\") pod \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " Jul 08 01:47:31.389144 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.388881 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-service-ca\") pod \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " Jul 08 01:47:31.389144 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.388906 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-trusted-ca-bundle\") pod \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " Jul 08 01:47:31.389144 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.388938 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-config\") pod \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\" (UID: \"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994\") " Jul 08 01:47:31.389324 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.389297 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-service-ca" (OuterVolumeSpecName: "service-ca") pod "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" (UID: "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:47:31.389379 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.389317 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" (UID: "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:47:31.389437 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.389402 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-config" (OuterVolumeSpecName: "console-config") pod "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" (UID: "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:47:31.389512 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.389435 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" (UID: "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:47:31.390984 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.390955 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" (UID: "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:47:31.390984 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.390967 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-kube-api-access-kh4fc" (OuterVolumeSpecName: "kube-api-access-kh4fc") pod "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" (UID: "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994"). InnerVolumeSpecName "kube-api-access-kh4fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:47:31.391114 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.390978 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" (UID: "ebf4c8a6-5e6b-4640-96b9-f09bfda0e994"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:47:31.490115 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.490085 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-oauth-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:47:31.490115 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.490110 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:47:31.490115 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.490122 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-service-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:47:31.490332 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.490131 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-trusted-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:47:31.490332 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.490139 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:47:31.490332 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.490148 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-console-oauth-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:47:31.490332 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:31.490157 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh4fc\" (UniqueName: \"kubernetes.io/projected/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994-kube-api-access-kh4fc\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:47:32.130546 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:32.130512 2578 generic.go:356] "Generic (PLEG): container finished" podID="ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" containerID="910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20" exitCode=0 Jul 08 01:47:32.130988 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:32.130555 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c88fcd5c-55mp6" event={"ID":"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994","Type":"ContainerDied","Data":"910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20"} Jul 08 01:47:32.130988 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:32.130593 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c88fcd5c-55mp6" Jul 08 01:47:32.130988 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:32.130603 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c88fcd5c-55mp6" event={"ID":"ebf4c8a6-5e6b-4640-96b9-f09bfda0e994","Type":"ContainerDied","Data":"2f433d50d7fc7804b1a402e138122a0602818c69bcc408c4e1d0785be44c107b"} Jul 08 01:47:32.130988 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:32.130631 2578 scope.go:117] "RemoveContainer" containerID="910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20" Jul 08 01:47:32.138597 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:32.138579 2578 scope.go:117] "RemoveContainer" containerID="910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20" Jul 08 01:47:32.138900 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:47:32.138878 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20\": container with ID starting with 910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20 not found: ID does not exist" containerID="910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20" Jul 08 01:47:32.138991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:32.138909 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20"} err="failed to get container status \"910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20\": rpc error: code = NotFound desc = could not find container \"910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20\": container with ID starting with 910a3f102ab9ef9979415009f83e25d1b3937396d5f950a3c1626a7673b9cc20 not found: ID does not exist" Jul 08 01:47:32.146613 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:32.146594 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c88fcd5c-55mp6"] Jul 08 01:47:32.150160 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:32.150140 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c88fcd5c-55mp6"] Jul 08 01:47:33.905517 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:33.905466 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" path="/var/lib/kubelet/pods/ebf4c8a6-5e6b-4640-96b9-f09bfda0e994/volumes" Jul 08 01:47:49.721209 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:47:49.721183 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 01:48:29.169550 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.169512 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4bgt2"] Jul 08 01:48:29.171837 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.169848 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" containerName="console" Jul 08 01:48:29.171837 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.169866 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" containerName="console" Jul 08 01:48:29.171837 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.169958 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf4c8a6-5e6b-4640-96b9-f09bfda0e994" containerName="console" Jul 08 01:48:29.172734 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.172718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4bgt2" Jul 08 01:48:29.175079 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.175058 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 01:48:29.175949 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.175912 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cm5zt\"" Jul 08 01:48:29.175949 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.175936 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 01:48:29.181249 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.181227 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4bgt2"] Jul 08 01:48:29.220763 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.220742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4pzn\" (UniqueName: \"kubernetes.io/projected/64222f95-af96-4994-bc15-b0f75682747a-kube-api-access-c4pzn\") pod \"cert-manager-55b4d954d5-4bgt2\" (UID: \"64222f95-af96-4994-bc15-b0f75682747a\") " pod="cert-manager/cert-manager-55b4d954d5-4bgt2" Jul 08 01:48:29.220869 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.220770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64222f95-af96-4994-bc15-b0f75682747a-bound-sa-token\") pod \"cert-manager-55b4d954d5-4bgt2\" (UID: \"64222f95-af96-4994-bc15-b0f75682747a\") " pod="cert-manager/cert-manager-55b4d954d5-4bgt2" Jul 08 01:48:29.321451 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.321420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4pzn\" (UniqueName: \"kubernetes.io/projected/64222f95-af96-4994-bc15-b0f75682747a-kube-api-access-c4pzn\") pod \"cert-manager-55b4d954d5-4bgt2\" (UID: \"64222f95-af96-4994-bc15-b0f75682747a\") " pod="cert-manager/cert-manager-55b4d954d5-4bgt2" Jul 08 01:48:29.321610 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.321454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64222f95-af96-4994-bc15-b0f75682747a-bound-sa-token\") pod \"cert-manager-55b4d954d5-4bgt2\" (UID: \"64222f95-af96-4994-bc15-b0f75682747a\") " pod="cert-manager/cert-manager-55b4d954d5-4bgt2" Jul 08 01:48:29.331386 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.331357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64222f95-af96-4994-bc15-b0f75682747a-bound-sa-token\") pod \"cert-manager-55b4d954d5-4bgt2\" (UID: \"64222f95-af96-4994-bc15-b0f75682747a\") " pod="cert-manager/cert-manager-55b4d954d5-4bgt2" Jul 08 01:48:29.331604 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.331583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4pzn\" (UniqueName: \"kubernetes.io/projected/64222f95-af96-4994-bc15-b0f75682747a-kube-api-access-c4pzn\") pod \"cert-manager-55b4d954d5-4bgt2\" (UID: \"64222f95-af96-4994-bc15-b0f75682747a\") " pod="cert-manager/cert-manager-55b4d954d5-4bgt2" Jul 08 01:48:29.492849 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.492794 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4bgt2" Jul 08 01:48:29.615787 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.615755 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4bgt2"] Jul 08 01:48:29.618998 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:48:29.618970 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64222f95_af96_4994_bc15_b0f75682747a.slice/crio-33fdf2246a311d5b0aa8b7a512f56f0d89f42d7717a0d30aaf574434e53e5690 WatchSource:0}: Error finding container 33fdf2246a311d5b0aa8b7a512f56f0d89f42d7717a0d30aaf574434e53e5690: Status 404 returned error can't find the container with id 33fdf2246a311d5b0aa8b7a512f56f0d89f42d7717a0d30aaf574434e53e5690 Jul 08 01:48:29.620953 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:29.620935 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 01:48:30.341271 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:30.341223 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4bgt2" event={"ID":"64222f95-af96-4994-bc15-b0f75682747a","Type":"ContainerStarted","Data":"33fdf2246a311d5b0aa8b7a512f56f0d89f42d7717a0d30aaf574434e53e5690"} Jul 08 01:48:33.354406 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:33.354369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4bgt2" event={"ID":"64222f95-af96-4994-bc15-b0f75682747a","Type":"ContainerStarted","Data":"5972ccea3b5ac7d85b5b92d8530d44ddcb007b3d1fa160e49c52dd57cb7cf8a0"} Jul 08 01:48:33.369892 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:33.369842 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4bgt2" podStartSLOduration=1.517072709 podStartE2EDuration="4.369827683s" totalImagesPullingTime="2.852754974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:48:33.368998641 +0000 UTC m=+344.030913179" watchObservedRunningTime="2026-07-08 01:48:33.369827683 +0000 UTC m=+344.031742220" Jul 08 01:48:50.305267 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.305188 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf"] Jul 08 01:48:50.318193 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.318164 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.321439 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.321417 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 01:48:50.321577 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.321495 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2dr79\"" Jul 08 01:48:50.321649 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.321624 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 01:48:50.321887 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.321868 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 01:48:50.322171 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.322158 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 01:48:50.329026 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.329007 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf"] Jul 08 01:48:50.384615 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.384592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c697a55-4ff5-42e1-9909-681e49531bab-manifests\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.384746 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.384619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c697a55-4ff5-42e1-9909-681e49531bab-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.384746 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.384638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c697a55-4ff5-42e1-9909-681e49531bab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.384746 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.384722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfg9k\" (UniqueName: \"kubernetes.io/projected/1c697a55-4ff5-42e1-9909-681e49531bab-kube-api-access-nfg9k\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.384844 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.384792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c697a55-4ff5-42e1-9909-681e49531bab-tmp\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.485746 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.485721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c697a55-4ff5-42e1-9909-681e49531bab-tmp\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.485894 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.485768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c697a55-4ff5-42e1-9909-681e49531bab-manifests\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.485894 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.485786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c697a55-4ff5-42e1-9909-681e49531bab-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.485894 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.485803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c697a55-4ff5-42e1-9909-681e49531bab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.485894 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.485849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfg9k\" (UniqueName: \"kubernetes.io/projected/1c697a55-4ff5-42e1-9909-681e49531bab-kube-api-access-nfg9k\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.486139 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.486118 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c697a55-4ff5-42e1-9909-681e49531bab-manifests\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.488033 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.488007 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c697a55-4ff5-42e1-9909-681e49531bab-tmp\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.488248 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.488228 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c697a55-4ff5-42e1-9909-681e49531bab-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.488288 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.488270 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c697a55-4ff5-42e1-9909-681e49531bab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.493557 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.493535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfg9k\" (UniqueName: \"kubernetes.io/projected/1c697a55-4ff5-42e1-9909-681e49531bab-kube-api-access-nfg9k\") pod \"opendatahub-operator-controller-manager-8bbd955f5-c77pf\" (UID: \"1c697a55-4ff5-42e1-9909-681e49531bab\") " pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.630101 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.630076 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:50.785462 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:50.785436 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf"] Jul 08 01:48:50.787890 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:48:50.787861 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c697a55_4ff5_42e1_9909_681e49531bab.slice/crio-0de10afa97811874b5584f2336296a2446d1b666982e9de0d2e7f0dd3dcb573e WatchSource:0}: Error finding container 0de10afa97811874b5584f2336296a2446d1b666982e9de0d2e7f0dd3dcb573e: Status 404 returned error can't find the container with id 0de10afa97811874b5584f2336296a2446d1b666982e9de0d2e7f0dd3dcb573e Jul 08 01:48:51.422634 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:51.422598 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" event={"ID":"1c697a55-4ff5-42e1-9909-681e49531bab","Type":"ContainerStarted","Data":"0de10afa97811874b5584f2336296a2446d1b666982e9de0d2e7f0dd3dcb573e"} Jul 08 01:48:54.436765 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:54.436727 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" event={"ID":"1c697a55-4ff5-42e1-9909-681e49531bab","Type":"ContainerStarted","Data":"3a5a7ade466cf379e1526793393f6a917303ef970269be5e4c6325cfe5744502"} Jul 08 01:48:55.442239 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:55.442207 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c697a55-4ff5-42e1-9909-681e49531bab" containerID="3a5a7ade466cf379e1526793393f6a917303ef970269be5e4c6325cfe5744502" exitCode=0 Jul 08 01:48:55.442722 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:55.442291 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" event={"ID":"1c697a55-4ff5-42e1-9909-681e49531bab","Type":"ContainerDied","Data":"3a5a7ade466cf379e1526793393f6a917303ef970269be5e4c6325cfe5744502"} Jul 08 01:48:56.448506 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:56.448451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" event={"ID":"1c697a55-4ff5-42e1-9909-681e49531bab","Type":"ContainerStarted","Data":"ecd616e149ca816427b5d0415e2a5addc8839219f2d04719b164ab8f7e8e97bc"} Jul 08 01:48:56.448868 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:56.448573 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:48:56.476856 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:48:56.476810 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" podStartSLOduration=2.063228441 podStartE2EDuration="6.476794451s" totalImagesPullingTime="3.33041523s" totalInitContainerRuntime="1.08315078s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:48:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:48:56.476043345 +0000 UTC m=+367.137957886" watchObservedRunningTime="2026-07-08 01:48:56.476794451 +0000 UTC m=+367.138709047" Jul 08 01:49:07.459138 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:07.459110 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8bbd955f5-c77pf" Jul 08 01:49:10.367255 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.367225 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57"] Jul 08 01:49:10.372825 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.372803 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.376278 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.376258 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 01:49:10.376521 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.376505 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 01:49:10.377107 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.377091 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 01:49:10.377203 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.377120 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 01:49:10.377267 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.377219 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bq7s5\"" Jul 08 01:49:10.377371 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.377356 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 01:49:10.383653 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.383633 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57"] Jul 08 01:49:10.449188 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.449160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfkq9\" (UniqueName: \"kubernetes.io/projected/c2ba6d26-f752-48ee-b4ba-bffa7df36502-kube-api-access-wfkq9\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.449324 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.449215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2ba6d26-f752-48ee-b4ba-bffa7df36502-metrics-cert\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.449324 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.449261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2ba6d26-f752-48ee-b4ba-bffa7df36502-manager-config\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.449324 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.449289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2ba6d26-f752-48ee-b4ba-bffa7df36502-cert\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.549943 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.549908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfkq9\" (UniqueName: \"kubernetes.io/projected/c2ba6d26-f752-48ee-b4ba-bffa7df36502-kube-api-access-wfkq9\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.550117 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.549967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2ba6d26-f752-48ee-b4ba-bffa7df36502-metrics-cert\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.550117 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.549994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2ba6d26-f752-48ee-b4ba-bffa7df36502-manager-config\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.550117 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.550025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2ba6d26-f752-48ee-b4ba-bffa7df36502-cert\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.550644 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.550619 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2ba6d26-f752-48ee-b4ba-bffa7df36502-manager-config\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.552378 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.552356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2ba6d26-f752-48ee-b4ba-bffa7df36502-metrics-cert\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.552490 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.552434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2ba6d26-f752-48ee-b4ba-bffa7df36502-cert\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.557633 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.557616 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfkq9\" (UniqueName: \"kubernetes.io/projected/c2ba6d26-f752-48ee-b4ba-bffa7df36502-kube-api-access-wfkq9\") pod \"lws-controller-manager-7769fc489f-xlh57\" (UID: \"c2ba6d26-f752-48ee-b4ba-bffa7df36502\") " pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.682194 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.682119 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:10.806729 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:10.806698 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57"] Jul 08 01:49:10.810028 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:49:10.809996 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ba6d26_f752_48ee_b4ba_bffa7df36502.slice/crio-449e97df8e52eefef3d9fd0abe4cd320983e120410bc7e7471278b374ee7c1e9 WatchSource:0}: Error finding container 449e97df8e52eefef3d9fd0abe4cd320983e120410bc7e7471278b374ee7c1e9: Status 404 returned error can't find the container with id 449e97df8e52eefef3d9fd0abe4cd320983e120410bc7e7471278b374ee7c1e9 Jul 08 01:49:11.509636 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:11.509598 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" event={"ID":"c2ba6d26-f752-48ee-b4ba-bffa7df36502","Type":"ContainerStarted","Data":"449e97df8e52eefef3d9fd0abe4cd320983e120410bc7e7471278b374ee7c1e9"} Jul 08 01:49:15.527820 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.527780 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" event={"ID":"c2ba6d26-f752-48ee-b4ba-bffa7df36502","Type":"ContainerStarted","Data":"a429e9b7f708afb1a2cd2c15d7e975ec986031b0485896b327296c2494aae854"} Jul 08 01:49:15.528239 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.528007 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:15.564245 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.564217 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84bf889db5-hbprh"] Jul 08 01:49:15.565852 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.565812 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" podStartSLOduration=1.864460061 podStartE2EDuration="5.565797994s" totalImagesPullingTime="3.701337933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:49:15.564791884 +0000 UTC m=+386.226706432" watchObservedRunningTime="2026-07-08 01:49:15.565797994 +0000 UTC m=+386.227712593" Jul 08 01:49:15.566887 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.566871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:15.569624 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.569605 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 01:49:15.569787 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.569663 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 01:49:15.569787 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.569735 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 01:49:15.570276 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.570261 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hp4c8\"" Jul 08 01:49:15.570373 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.570358 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 01:49:15.585676 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.585655 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84bf889db5-hbprh"] Jul 08 01:49:15.697604 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.697577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31844cd7-e4fc-4a12-9b76-88434d337770-tls-certs\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:15.697756 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.697731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31844cd7-e4fc-4a12-9b76-88434d337770-tmp\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:15.697818 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.697776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7hqd\" (UniqueName: \"kubernetes.io/projected/31844cd7-e4fc-4a12-9b76-88434d337770-kube-api-access-g7hqd\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:15.798967 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.798911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31844cd7-e4fc-4a12-9b76-88434d337770-tmp\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:15.798967 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.798945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7hqd\" (UniqueName: \"kubernetes.io/projected/31844cd7-e4fc-4a12-9b76-88434d337770-kube-api-access-g7hqd\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:15.799107 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.798980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31844cd7-e4fc-4a12-9b76-88434d337770-tls-certs\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:15.799107 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:49:15.799075 2578 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 08 01:49:15.799183 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:49:15.799120 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31844cd7-e4fc-4a12-9b76-88434d337770-tls-certs podName:31844cd7-e4fc-4a12-9b76-88434d337770 nodeName:}" failed. No retries permitted until 2026-07-08 01:49:16.299105385 +0000 UTC m=+386.961019900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/31844cd7-e4fc-4a12-9b76-88434d337770-tls-certs") pod "kube-auth-proxy-84bf889db5-hbprh" (UID: "31844cd7-e4fc-4a12-9b76-88434d337770") : secret "kube-auth-proxy-tls" not found Jul 08 01:49:15.801076 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.801046 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31844cd7-e4fc-4a12-9b76-88434d337770-tmp\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:15.807208 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:15.807187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7hqd\" (UniqueName: \"kubernetes.io/projected/31844cd7-e4fc-4a12-9b76-88434d337770-kube-api-access-g7hqd\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:16.302884 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:16.302845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31844cd7-e4fc-4a12-9b76-88434d337770-tls-certs\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:16.305222 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:16.305203 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31844cd7-e4fc-4a12-9b76-88434d337770-tls-certs\") pod \"kube-auth-proxy-84bf889db5-hbprh\" (UID: \"31844cd7-e4fc-4a12-9b76-88434d337770\") " pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:16.475513 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:16.475463 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" Jul 08 01:49:16.608830 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:16.608804 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84bf889db5-hbprh"] Jul 08 01:49:16.611775 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:49:16.611746 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31844cd7_e4fc_4a12_9b76_88434d337770.slice/crio-5c402d7bc2e6aa8d22550e9065387a9e237e24ce5814e61319f1aad3120c7f6b WatchSource:0}: Error finding container 5c402d7bc2e6aa8d22550e9065387a9e237e24ce5814e61319f1aad3120c7f6b: Status 404 returned error can't find the container with id 5c402d7bc2e6aa8d22550e9065387a9e237e24ce5814e61319f1aad3120c7f6b Jul 08 01:49:17.539114 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:17.539073 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" event={"ID":"31844cd7-e4fc-4a12-9b76-88434d337770","Type":"ContainerStarted","Data":"5c402d7bc2e6aa8d22550e9065387a9e237e24ce5814e61319f1aad3120c7f6b"} Jul 08 01:49:19.548947 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:19.548915 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" event={"ID":"31844cd7-e4fc-4a12-9b76-88434d337770","Type":"ContainerStarted","Data":"fbac87aac5d57630879655a43015f973a7082eac65fe8a02e81c04d2b968348c"} Jul 08 01:49:19.570113 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:19.570072 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84bf889db5-hbprh" podStartSLOduration=1.709294936 podStartE2EDuration="4.570058409s" totalImagesPullingTime="2.860763473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:49:19.568233407 +0000 UTC m=+390.230147945" watchObservedRunningTime="2026-07-08 01:49:19.570058409 +0000 UTC m=+390.231972946" Jul 08 01:49:20.610021 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.609989 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4"] Jul 08 01:49:20.612371 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.612349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:20.615229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.615204 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-sz7ts\"" Jul 08 01:49:20.615340 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.615275 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 08 01:49:20.631392 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.631370 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4"] Jul 08 01:49:20.740497 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.740445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhmw\" (UniqueName: \"kubernetes.io/projected/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-kube-api-access-jbhmw\") pod \"kserve-controller-manager-d6f4b65b8-m2bv4\" (UID: \"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:20.740647 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.740569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-cert\") pod \"kserve-controller-manager-d6f4b65b8-m2bv4\" (UID: \"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:20.841585 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.841554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-cert\") pod \"kserve-controller-manager-d6f4b65b8-m2bv4\" (UID: \"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:20.841713 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.841618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhmw\" (UniqueName: \"kubernetes.io/projected/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-kube-api-access-jbhmw\") pod \"kserve-controller-manager-d6f4b65b8-m2bv4\" (UID: \"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:20.841713 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:49:20.841673 2578 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 08 01:49:20.841816 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:49:20.841727 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-cert podName:bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf nodeName:}" failed. No retries permitted until 2026-07-08 01:49:21.341712838 +0000 UTC m=+392.003627357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-cert") pod "kserve-controller-manager-d6f4b65b8-m2bv4" (UID: "bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf") : secret "kserve-webhook-server-cert" not found Jul 08 01:49:20.855488 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:20.855460 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhmw\" (UniqueName: \"kubernetes.io/projected/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-kube-api-access-jbhmw\") pod \"kserve-controller-manager-d6f4b65b8-m2bv4\" (UID: \"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:21.346973 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:21.346925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-cert\") pod \"kserve-controller-manager-d6f4b65b8-m2bv4\" (UID: \"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:21.347180 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:49:21.347099 2578 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 08 01:49:21.347180 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:49:21.347173 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-cert podName:bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf nodeName:}" failed. No retries permitted until 2026-07-08 01:49:22.347154291 +0000 UTC m=+393.009068810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-cert") pod "kserve-controller-manager-d6f4b65b8-m2bv4" (UID: "bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf") : secret "kserve-webhook-server-cert" not found Jul 08 01:49:22.354595 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.354559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-cert\") pod \"kserve-controller-manager-d6f4b65b8-m2bv4\" (UID: \"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:22.356877 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.356856 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf-cert\") pod \"kserve-controller-manager-d6f4b65b8-m2bv4\" (UID: \"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:22.421927 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.421892 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:22.550695 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.550521 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4"] Jul 08 01:49:22.553658 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:49:22.553630 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf78b36_1ffe_4646_b9b3_c3ecabf5eecf.slice/crio-9d4e25aeb18638f23222623494f9b732f99eb9604d5306a43a9fe50fae26df21 WatchSource:0}: Error finding container 9d4e25aeb18638f23222623494f9b732f99eb9604d5306a43a9fe50fae26df21: Status 404 returned error can't find the container with id 9d4e25aeb18638f23222623494f9b732f99eb9604d5306a43a9fe50fae26df21 Jul 08 01:49:22.563760 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.563731 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" event={"ID":"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf","Type":"ContainerStarted","Data":"9d4e25aeb18638f23222623494f9b732f99eb9604d5306a43a9fe50fae26df21"} Jul 08 01:49:22.790400 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.790328 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j"] Jul 08 01:49:22.807227 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.807208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:22.810533 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.810514 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 01:49:22.810741 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.810726 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-72r5b\"" Jul 08 01:49:22.816366 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.816347 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j"] Jul 08 01:49:22.958133 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.958095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:22.958311 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.958178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8kz6\" (UniqueName: \"kubernetes.io/projected/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-kube-api-access-j8kz6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:22.958311 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.958230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:22.958426 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.958338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:22.958426 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.958387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:22.958560 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.958448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:22.958560 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.958500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:22.958560 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.958524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:22.958560 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:22.958549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.059629 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.059560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8kz6\" (UniqueName: \"kubernetes.io/projected/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-kube-api-access-j8kz6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.059629 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.059606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.059834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.059646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.059834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.059683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.059834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.059721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.059834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.059746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.059834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.059773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.059834 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.059804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.060094 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.059837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.060156 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.060092 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.060225 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.060149 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.060279 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.060222 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.060279 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.060262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.060546 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.060523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.062556 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.062532 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.062736 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.062718 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.071628 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.071284 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.071628 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.071586 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8kz6\" (UniqueName: \"kubernetes.io/projected/cebdefad-b4d7-4753-a0ea-b9e4e081db8e-kube-api-access-j8kz6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zhm6j\" (UID: \"cebdefad-b4d7-4753-a0ea-b9e4e081db8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.118661 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.118635 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:23.283008 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.282974 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j"] Jul 08 01:49:23.289592 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:49:23.289562 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebdefad_b4d7_4753_a0ea_b9e4e081db8e.slice/crio-264acfad9f141b2fb413ca0657005939e73ae2d79714d9d582786658e2fdb963 WatchSource:0}: Error finding container 264acfad9f141b2fb413ca0657005939e73ae2d79714d9d582786658e2fdb963: Status 404 returned error can't find the container with id 264acfad9f141b2fb413ca0657005939e73ae2d79714d9d582786658e2fdb963 Jul 08 01:49:23.569545 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:23.569508 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" event={"ID":"cebdefad-b4d7-4753-a0ea-b9e4e081db8e","Type":"ContainerStarted","Data":"264acfad9f141b2fb413ca0657005939e73ae2d79714d9d582786658e2fdb963"} Jul 08 01:49:25.581643 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:25.581552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" event={"ID":"bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf","Type":"ContainerStarted","Data":"3f384e7085843d0aaa16c7a366e167d4105c44b03f8a1b80d9a665296c947745"} Jul 08 01:49:25.582065 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:25.581693 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:49:25.599512 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:25.599451 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" podStartSLOduration=2.901134603 podStartE2EDuration="5.599436439s" totalImagesPullingTime="2.698301836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:49:25.59744152 +0000 UTC m=+396.259356070" watchObservedRunningTime="2026-07-08 01:49:25.599436439 +0000 UTC m=+396.261350976" Jul 08 01:49:26.534425 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:26.534396 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7769fc489f-xlh57" Jul 08 01:49:26.794740 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:26.794698 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 01:49:26.795021 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:26.794779 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 01:49:26.795021 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:26.794809 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 01:49:27.593253 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:27.593221 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" event={"ID":"cebdefad-b4d7-4753-a0ea-b9e4e081db8e","Type":"ContainerStarted","Data":"a8ea475e3e7a9ed04456015d32d6e70a076d33343a376139b8ffc410241a7cd8"} Jul 08 01:49:27.617068 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:27.617021 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" podStartSLOduration=2.115205079 podStartE2EDuration="5.617005755s" totalImagesPullingTime="3.501800676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:49:27.615727747 +0000 UTC m=+398.277642285" watchObservedRunningTime="2026-07-08 01:49:27.617005755 +0000 UTC m=+398.278920293" Jul 08 01:49:28.119626 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:28.119599 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:28.123857 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:28.123834 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:28.597991 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:28.597960 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:28.598809 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:28.598787 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zhm6j" Jul 08 01:49:56.592513 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:49:56.592464 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m2bv4" Jul 08 01:50:17.848027 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:17.847951 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58f6c9449-d479d"] Jul 08 01:50:17.851134 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:17.851112 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:17.863901 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:17.863876 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f6c9449-d479d"] Jul 08 01:50:18.017169 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.017131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-console-config\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.017342 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.017217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-oauth-serving-cert\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.017342 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.017241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-trusted-ca-bundle\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.017342 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.017267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f06e966a-cd7f-43b0-80d0-ba771ba448fb-console-serving-cert\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.017342 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.017293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f06e966a-cd7f-43b0-80d0-ba771ba448fb-console-oauth-config\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.017504 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.017373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxpj\" (UniqueName: \"kubernetes.io/projected/f06e966a-cd7f-43b0-80d0-ba771ba448fb-kube-api-access-5dxpj\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.017504 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.017416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-service-ca\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.118316 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.118231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-console-config\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.118316 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.118289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-oauth-serving-cert\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.118316 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.118314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-trusted-ca-bundle\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.118621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.118334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f06e966a-cd7f-43b0-80d0-ba771ba448fb-console-serving-cert\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.118621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.118358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f06e966a-cd7f-43b0-80d0-ba771ba448fb-console-oauth-config\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.118621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.118390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxpj\" (UniqueName: \"kubernetes.io/projected/f06e966a-cd7f-43b0-80d0-ba771ba448fb-kube-api-access-5dxpj\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.118621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.118417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-service-ca\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.119139 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.119111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-console-config\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.119229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.119115 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-oauth-serving-cert\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.119229 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.119210 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-service-ca\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.119514 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.119469 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f06e966a-cd7f-43b0-80d0-ba771ba448fb-trusted-ca-bundle\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.120888 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.120865 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f06e966a-cd7f-43b0-80d0-ba771ba448fb-console-oauth-config\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.120967 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.120922 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f06e966a-cd7f-43b0-80d0-ba771ba448fb-console-serving-cert\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.125798 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.125779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxpj\" (UniqueName: \"kubernetes.io/projected/f06e966a-cd7f-43b0-80d0-ba771ba448fb-kube-api-access-5dxpj\") pod \"console-58f6c9449-d479d\" (UID: \"f06e966a-cd7f-43b0-80d0-ba771ba448fb\") " pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.160994 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.160968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:18.292237 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.292162 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f6c9449-d479d"] Jul 08 01:50:18.294927 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:50:18.294899 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06e966a_cd7f_43b0_80d0_ba771ba448fb.slice/crio-82ffc053b6c3d88eae43fd65f8bb9714183ad18dd0604981b71ff8171b7530c4 WatchSource:0}: Error finding container 82ffc053b6c3d88eae43fd65f8bb9714183ad18dd0604981b71ff8171b7530c4: Status 404 returned error can't find the container with id 82ffc053b6c3d88eae43fd65f8bb9714183ad18dd0604981b71ff8171b7530c4 Jul 08 01:50:18.808109 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.808073 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f6c9449-d479d" event={"ID":"f06e966a-cd7f-43b0-80d0-ba771ba448fb","Type":"ContainerStarted","Data":"d8af13180ed2536e2f9bdc6eb784ede37f8d3d652d9e4029cb569f2894b06880"} Jul 08 01:50:18.808297 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.808127 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f6c9449-d479d" event={"ID":"f06e966a-cd7f-43b0-80d0-ba771ba448fb","Type":"ContainerStarted","Data":"82ffc053b6c3d88eae43fd65f8bb9714183ad18dd0604981b71ff8171b7530c4"} Jul 08 01:50:18.826306 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:18.826258 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58f6c9449-d479d" podStartSLOduration=1.826243436 podStartE2EDuration="1.826243436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:50:18.824854639 +0000 UTC m=+449.486769173" watchObservedRunningTime="2026-07-08 01:50:18.826243436 +0000 UTC m=+449.488157973" Jul 08 01:50:28.161782 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:28.161745 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:28.162150 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:28.161814 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:28.166685 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:28.166664 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:28.855420 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:28.855393 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58f6c9449-d479d" Jul 08 01:50:28.904461 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:28.904424 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f88f7fc9d-tkzln"] Jul 08 01:50:53.924558 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:53.924502 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-f88f7fc9d-tkzln" podUID="64fc3648-ced9-44e4-b3a8-612561c26ebb" containerName="console" containerID="cri-o://916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed" gracePeriod=15 Jul 08 01:50:54.169178 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.169149 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:50:54.310714 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.310635 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-oauth-config\") pod \"64fc3648-ced9-44e4-b3a8-612561c26ebb\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " Jul 08 01:50:54.310714 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.310675 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-oauth-serving-cert\") pod \"64fc3648-ced9-44e4-b3a8-612561c26ebb\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " Jul 08 01:50:54.310714 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.310700 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-config\") pod \"64fc3648-ced9-44e4-b3a8-612561c26ebb\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " Jul 08 01:50:54.310951 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.310725 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-serving-cert\") pod \"64fc3648-ced9-44e4-b3a8-612561c26ebb\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " Jul 08 01:50:54.310951 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.310814 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w9zs\" (UniqueName: \"kubernetes.io/projected/64fc3648-ced9-44e4-b3a8-612561c26ebb-kube-api-access-2w9zs\") pod \"64fc3648-ced9-44e4-b3a8-612561c26ebb\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " Jul 08 01:50:54.310951 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.310861 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-trusted-ca-bundle\") pod \"64fc3648-ced9-44e4-b3a8-612561c26ebb\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " Jul 08 01:50:54.310951 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.310897 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-service-ca\") pod \"64fc3648-ced9-44e4-b3a8-612561c26ebb\" (UID: \"64fc3648-ced9-44e4-b3a8-612561c26ebb\") " Jul 08 01:50:54.311212 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.311068 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "64fc3648-ced9-44e4-b3a8-612561c26ebb" (UID: "64fc3648-ced9-44e4-b3a8-612561c26ebb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:50:54.311293 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.311211 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-config" (OuterVolumeSpecName: "console-config") pod "64fc3648-ced9-44e4-b3a8-612561c26ebb" (UID: "64fc3648-ced9-44e4-b3a8-612561c26ebb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:50:54.311293 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.311266 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "64fc3648-ced9-44e4-b3a8-612561c26ebb" (UID: "64fc3648-ced9-44e4-b3a8-612561c26ebb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:50:54.311447 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.311362 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-service-ca" (OuterVolumeSpecName: "service-ca") pod "64fc3648-ced9-44e4-b3a8-612561c26ebb" (UID: "64fc3648-ced9-44e4-b3a8-612561c26ebb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 01:50:54.312962 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.312936 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "64fc3648-ced9-44e4-b3a8-612561c26ebb" (UID: "64fc3648-ced9-44e4-b3a8-612561c26ebb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:50:54.313078 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.312995 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fc3648-ced9-44e4-b3a8-612561c26ebb-kube-api-access-2w9zs" (OuterVolumeSpecName: "kube-api-access-2w9zs") pod "64fc3648-ced9-44e4-b3a8-612561c26ebb" (UID: "64fc3648-ced9-44e4-b3a8-612561c26ebb"). InnerVolumeSpecName "kube-api-access-2w9zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:50:54.313078 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.313035 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "64fc3648-ced9-44e4-b3a8-612561c26ebb" (UID: "64fc3648-ced9-44e4-b3a8-612561c26ebb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:50:54.412322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.412292 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w9zs\" (UniqueName: \"kubernetes.io/projected/64fc3648-ced9-44e4-b3a8-612561c26ebb-kube-api-access-2w9zs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:50:54.412322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.412320 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-trusted-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:50:54.412322 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.412330 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-service-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:50:54.412566 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.412338 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-oauth-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:50:54.412566 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.412347 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-oauth-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:50:54.412566 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.412355 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:50:54.412566 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.412362 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64fc3648-ced9-44e4-b3a8-612561c26ebb-console-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:50:54.962245 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.962210 2578 generic.go:356] "Generic (PLEG): container finished" podID="64fc3648-ced9-44e4-b3a8-612561c26ebb" containerID="916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed" exitCode=0 Jul 08 01:50:54.962647 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.962258 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f88f7fc9d-tkzln" Jul 08 01:50:54.962647 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.962267 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f88f7fc9d-tkzln" event={"ID":"64fc3648-ced9-44e4-b3a8-612561c26ebb","Type":"ContainerDied","Data":"916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed"} Jul 08 01:50:54.962647 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.962315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f88f7fc9d-tkzln" event={"ID":"64fc3648-ced9-44e4-b3a8-612561c26ebb","Type":"ContainerDied","Data":"13138f91c6c32ef7a9774422aa3b72b6a9d8f14bd3d5cab32d37185c656c362f"} Jul 08 01:50:54.962647 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.962336 2578 scope.go:117] "RemoveContainer" containerID="916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed" Jul 08 01:50:54.970716 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.970694 2578 scope.go:117] "RemoveContainer" containerID="916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed" Jul 08 01:50:54.970974 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:50:54.970955 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed\": container with ID starting with 916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed not found: ID does not exist" containerID="916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed" Jul 08 01:50:54.971017 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.970984 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed"} err="failed to get container status \"916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed\": rpc error: code = NotFound desc = could not find container \"916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed\": container with ID starting with 916ce1303c2b85143b1a7a8674757893cd4d92af4a2b11017fae36092bb00fed not found: ID does not exist" Jul 08 01:50:54.983118 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.983095 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f88f7fc9d-tkzln"] Jul 08 01:50:54.989929 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:54.989898 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f88f7fc9d-tkzln"] Jul 08 01:50:55.905941 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:50:55.905908 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fc3648-ced9-44e4-b3a8-612561c26ebb" path="/var/lib/kubelet/pods/64fc3648-ced9-44e4-b3a8-612561c26ebb/volumes" Jul 08 01:51:02.117433 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.117399 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k"] Jul 08 01:51:02.117797 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.117727 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fc3648-ced9-44e4-b3a8-612561c26ebb" containerName="console" Jul 08 01:51:02.117797 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.117738 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fc3648-ced9-44e4-b3a8-612561c26ebb" containerName="console" Jul 08 01:51:02.117879 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.117804 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fc3648-ced9-44e4-b3a8-612561c26ebb" containerName="console" Jul 08 01:51:02.122400 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.122379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.124812 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.124791 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 08 01:51:02.124944 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.124924 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 08 01:51:02.125010 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.124928 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 01:51:02.125778 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.125757 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-27b8d\"" Jul 08 01:51:02.125893 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.125796 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 01:51:02.131189 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.131172 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k"] Jul 08 01:51:02.279496 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.279452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e6a88fd-1494-4658-bb29-58c2370ed927-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.279644 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.279560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e6a88fd-1494-4658-bb29-58c2370ed927-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.279644 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.279595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db8lx\" (UniqueName: \"kubernetes.io/projected/5e6a88fd-1494-4658-bb29-58c2370ed927-kube-api-access-db8lx\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.380305 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.380233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e6a88fd-1494-4658-bb29-58c2370ed927-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.380305 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.380291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e6a88fd-1494-4658-bb29-58c2370ed927-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.380534 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.380316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db8lx\" (UniqueName: \"kubernetes.io/projected/5e6a88fd-1494-4658-bb29-58c2370ed927-kube-api-access-db8lx\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.380534 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:51:02.380389 2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 08 01:51:02.380534 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:51:02.380472 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6a88fd-1494-4658-bb29-58c2370ed927-plugin-serving-cert podName:5e6a88fd-1494-4658-bb29-58c2370ed927 nodeName:}" failed. No retries permitted until 2026-07-08 01:51:02.880451337 +0000 UTC m=+493.542365857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/5e6a88fd-1494-4658-bb29-58c2370ed927-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-vzw5k" (UID: "5e6a88fd-1494-4658-bb29-58c2370ed927") : secret "plugin-serving-cert" not found Jul 08 01:51:02.380967 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.380948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e6a88fd-1494-4658-bb29-58c2370ed927-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.389968 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.389949 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db8lx\" (UniqueName: \"kubernetes.io/projected/5e6a88fd-1494-4658-bb29-58c2370ed927-kube-api-access-db8lx\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.884756 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.884724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e6a88fd-1494-4658-bb29-58c2370ed927-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:02.887032 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:02.887013 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e6a88fd-1494-4658-bb29-58c2370ed927-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vzw5k\" (UID: \"5e6a88fd-1494-4658-bb29-58c2370ed927\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:03.032592 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:03.032567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" Jul 08 01:51:03.160030 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:03.159966 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k"] Jul 08 01:51:03.163411 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:51:03.163385 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e6a88fd_1494_4658_bb29_58c2370ed927.slice/crio-399030db6d01ce64ed5263bbcc3328bc285a8162ea202370170b1a41bbe09715 WatchSource:0}: Error finding container 399030db6d01ce64ed5263bbcc3328bc285a8162ea202370170b1a41bbe09715: Status 404 returned error can't find the container with id 399030db6d01ce64ed5263bbcc3328bc285a8162ea202370170b1a41bbe09715 Jul 08 01:51:04.003026 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:04.002995 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" event={"ID":"5e6a88fd-1494-4658-bb29-58c2370ed927","Type":"ContainerStarted","Data":"399030db6d01ce64ed5263bbcc3328bc285a8162ea202370170b1a41bbe09715"} Jul 08 01:51:28.122279 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:28.122243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" event={"ID":"5e6a88fd-1494-4658-bb29-58c2370ed927","Type":"ContainerStarted","Data":"7264eb44e5742a265a790fad4a3b5be6d4420c82968b65985df947d7e5f06ba4"} Jul 08 01:51:28.141226 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:28.141183 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vzw5k" podStartSLOduration=1.49056165 podStartE2EDuration="26.141169029s" totalImagesPullingTime="24.650607379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:51:28.138809144 +0000 UTC m=+518.800723681" watchObservedRunningTime="2026-07-08 01:51:28.141169029 +0000 UTC m=+518.803083609" Jul 08 01:51:51.297056 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.297027 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:51:51.301346 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.301323 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:51:51.303641 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.303620 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 01:51:51.310313 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.310290 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:51:51.344381 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.344352 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:51:51.419536 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.419511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4fmr\" (UniqueName: \"kubernetes.io/projected/9dee3815-0b23-4a22-851d-10b7b1ea70aa-kube-api-access-z4fmr\") pod \"limitador-limitador-54d6d988d5-jvbzf\" (UID: \"9dee3815-0b23-4a22-851d-10b7b1ea70aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:51:51.419632 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.419561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9dee3815-0b23-4a22-851d-10b7b1ea70aa-config-file\") pod \"limitador-limitador-54d6d988d5-jvbzf\" (UID: \"9dee3815-0b23-4a22-851d-10b7b1ea70aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:51:51.520564 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.520517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9dee3815-0b23-4a22-851d-10b7b1ea70aa-config-file\") pod \"limitador-limitador-54d6d988d5-jvbzf\" (UID: \"9dee3815-0b23-4a22-851d-10b7b1ea70aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:51:51.520705 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.520635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fmr\" (UniqueName: \"kubernetes.io/projected/9dee3815-0b23-4a22-851d-10b7b1ea70aa-kube-api-access-z4fmr\") pod \"limitador-limitador-54d6d988d5-jvbzf\" (UID: \"9dee3815-0b23-4a22-851d-10b7b1ea70aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:51:51.521163 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.521142 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9dee3815-0b23-4a22-851d-10b7b1ea70aa-config-file\") pod \"limitador-limitador-54d6d988d5-jvbzf\" (UID: \"9dee3815-0b23-4a22-851d-10b7b1ea70aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:51:51.529034 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.529013 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fmr\" (UniqueName: \"kubernetes.io/projected/9dee3815-0b23-4a22-851d-10b7b1ea70aa-kube-api-access-z4fmr\") pod \"limitador-limitador-54d6d988d5-jvbzf\" (UID: \"9dee3815-0b23-4a22-851d-10b7b1ea70aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:51:51.611406 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.611380 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:51:51.755882 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.755854 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:51:51.758132 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:51:51.758105 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dee3815_0b23_4a22_851d_10b7b1ea70aa.slice/crio-5088b5e27f93858743cd23f0a2dac78c6dc32de72d2c5325cbe4200b2fbea227 WatchSource:0}: Error finding container 5088b5e27f93858743cd23f0a2dac78c6dc32de72d2c5325cbe4200b2fbea227: Status 404 returned error can't find the container with id 5088b5e27f93858743cd23f0a2dac78c6dc32de72d2c5325cbe4200b2fbea227 Jul 08 01:51:51.896022 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.895950 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-t8scl"] Jul 08 01:51:51.900160 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.900139 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t8scl" Jul 08 01:51:51.902388 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.902366 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-57448\"" Jul 08 01:51:51.906923 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:51.906902 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t8scl"] Jul 08 01:51:52.024469 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:52.024437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwk8n\" (UniqueName: \"kubernetes.io/projected/4d3eb237-590b-4b8c-8f22-794491364809-kube-api-access-cwk8n\") pod \"authorino-75d556bd77-t8scl\" (UID: \"4d3eb237-590b-4b8c-8f22-794491364809\") " pod="kuadrant-system/authorino-75d556bd77-t8scl" Jul 08 01:51:52.125225 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:52.125198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwk8n\" (UniqueName: \"kubernetes.io/projected/4d3eb237-590b-4b8c-8f22-794491364809-kube-api-access-cwk8n\") pod \"authorino-75d556bd77-t8scl\" (UID: \"4d3eb237-590b-4b8c-8f22-794491364809\") " pod="kuadrant-system/authorino-75d556bd77-t8scl" Jul 08 01:51:52.133405 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:52.133370 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwk8n\" (UniqueName: \"kubernetes.io/projected/4d3eb237-590b-4b8c-8f22-794491364809-kube-api-access-cwk8n\") pod \"authorino-75d556bd77-t8scl\" (UID: \"4d3eb237-590b-4b8c-8f22-794491364809\") " pod="kuadrant-system/authorino-75d556bd77-t8scl" Jul 08 01:51:52.211710 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:52.211640 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t8scl" Jul 08 01:51:52.232908 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:52.232877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" event={"ID":"9dee3815-0b23-4a22-851d-10b7b1ea70aa","Type":"ContainerStarted","Data":"5088b5e27f93858743cd23f0a2dac78c6dc32de72d2c5325cbe4200b2fbea227"} Jul 08 01:51:52.385838 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:52.385737 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t8scl"] Jul 08 01:51:52.389733 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:51:52.389698 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d3eb237_590b_4b8c_8f22_794491364809.slice/crio-6a06742f557dd02f67e80c493003bcff5ade0468258dfcc0d8c7936ec15f3477 WatchSource:0}: Error finding container 6a06742f557dd02f67e80c493003bcff5ade0468258dfcc0d8c7936ec15f3477: Status 404 returned error can't find the container with id 6a06742f557dd02f67e80c493003bcff5ade0468258dfcc0d8c7936ec15f3477 Jul 08 01:51:53.239453 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:53.239407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t8scl" event={"ID":"4d3eb237-590b-4b8c-8f22-794491364809","Type":"ContainerStarted","Data":"6a06742f557dd02f67e80c493003bcff5ade0468258dfcc0d8c7936ec15f3477"} Jul 08 01:51:55.252854 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:55.252816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" event={"ID":"9dee3815-0b23-4a22-851d-10b7b1ea70aa","Type":"ContainerStarted","Data":"8897aec41568f576b2b506728010b4829357af8c354e11492fd711bbc61d358c"} Jul 08 01:51:55.253306 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:55.252932 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:51:55.269835 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:55.269778 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" podStartSLOduration=1.741576083 podStartE2EDuration="4.269759383s" totalImagesPullingTime="2.5281833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:51:55.269256203 +0000 UTC m=+545.931170741" watchObservedRunningTime="2026-07-08 01:51:55.269759383 +0000 UTC m=+545.931673922" Jul 08 01:51:57.263520 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:57.263467 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t8scl" event={"ID":"4d3eb237-590b-4b8c-8f22-794491364809","Type":"ContainerStarted","Data":"3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216"} Jul 08 01:51:57.278691 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:51:57.278644 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-t8scl" podStartSLOduration=2.187171545 podStartE2EDuration="6.27862855s" totalImagesPullingTime="4.091457005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:51:57.277531446 +0000 UTC m=+547.939445984" watchObservedRunningTime="2026-07-08 01:51:57.27862855 +0000 UTC m=+547.940543088" Jul 08 01:52:06.259222 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:06.259192 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jvbzf" Jul 08 01:52:24.650131 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.650100 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vwb5m"] Jul 08 01:52:24.655006 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.654984 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" Jul 08 01:52:24.658382 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.658360 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vwb5m"] Jul 08 01:52:24.708729 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.708700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcm88\" (UniqueName: \"kubernetes.io/projected/9ea16ad4-aac0-4f03-97af-5289530b3ca9-kube-api-access-vcm88\") pod \"authorino-5d4b67c446-vwb5m\" (UID: \"9ea16ad4-aac0-4f03-97af-5289530b3ca9\") " pod="kuadrant-system/authorino-5d4b67c446-vwb5m" Jul 08 01:52:24.809193 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.809169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcm88\" (UniqueName: \"kubernetes.io/projected/9ea16ad4-aac0-4f03-97af-5289530b3ca9-kube-api-access-vcm88\") pod \"authorino-5d4b67c446-vwb5m\" (UID: \"9ea16ad4-aac0-4f03-97af-5289530b3ca9\") " pod="kuadrant-system/authorino-5d4b67c446-vwb5m" Jul 08 01:52:24.817593 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.817568 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcm88\" (UniqueName: \"kubernetes.io/projected/9ea16ad4-aac0-4f03-97af-5289530b3ca9-kube-api-access-vcm88\") pod \"authorino-5d4b67c446-vwb5m\" (UID: \"9ea16ad4-aac0-4f03-97af-5289530b3ca9\") " pod="kuadrant-system/authorino-5d4b67c446-vwb5m" Jul 08 01:52:24.902829 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.902772 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vwb5m"] Jul 08 01:52:24.903042 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.903030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" Jul 08 01:52:24.935643 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.935579 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66d74db596-b8ndd"] Jul 08 01:52:24.938113 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.938098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66d74db596-b8ndd" Jul 08 01:52:24.946198 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:24.946174 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66d74db596-b8ndd"] Jul 08 01:52:25.011230 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.011195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkd48\" (UniqueName: \"kubernetes.io/projected/4e83845f-1b24-409e-8ba5-d7eb9a4b8b29-kube-api-access-fkd48\") pod \"authorino-66d74db596-b8ndd\" (UID: \"4e83845f-1b24-409e-8ba5-d7eb9a4b8b29\") " pod="kuadrant-system/authorino-66d74db596-b8ndd" Jul 08 01:52:25.036162 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.035440 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vwb5m"] Jul 08 01:52:25.112080 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.112041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkd48\" (UniqueName: \"kubernetes.io/projected/4e83845f-1b24-409e-8ba5-d7eb9a4b8b29-kube-api-access-fkd48\") pod \"authorino-66d74db596-b8ndd\" (UID: \"4e83845f-1b24-409e-8ba5-d7eb9a4b8b29\") " pod="kuadrant-system/authorino-66d74db596-b8ndd" Jul 08 01:52:25.119938 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.119919 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkd48\" (UniqueName: \"kubernetes.io/projected/4e83845f-1b24-409e-8ba5-d7eb9a4b8b29-kube-api-access-fkd48\") pod \"authorino-66d74db596-b8ndd\" (UID: \"4e83845f-1b24-409e-8ba5-d7eb9a4b8b29\") " pod="kuadrant-system/authorino-66d74db596-b8ndd" Jul 08 01:52:25.249429 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.249353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66d74db596-b8ndd" Jul 08 01:52:25.251431 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.251303 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66d74db596-b8ndd"] Jul 08 01:52:25.281029 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.281001 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7957c9d8b8-fxc9l"] Jul 08 01:52:25.283644 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.283628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:52:25.286331 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.286314 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 08 01:52:25.294424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.294400 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7957c9d8b8-fxc9l"] Jul 08 01:52:25.374788 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.374753 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66d74db596-b8ndd"] Jul 08 01:52:25.376701 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:52:25.376672 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e83845f_1b24_409e_8ba5_d7eb9a4b8b29.slice/crio-927f5cc21582c0f7d1f04653aae122cff2e7696f94d31c6f39ef6412ec48b203 WatchSource:0}: Error finding container 927f5cc21582c0f7d1f04653aae122cff2e7696f94d31c6f39ef6412ec48b203: Status 404 returned error can't find the container with id 927f5cc21582c0f7d1f04653aae122cff2e7696f94d31c6f39ef6412ec48b203 Jul 08 01:52:25.390825 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.390796 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" event={"ID":"9ea16ad4-aac0-4f03-97af-5289530b3ca9","Type":"ContainerStarted","Data":"370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806"} Jul 08 01:52:25.390941 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.390841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" event={"ID":"9ea16ad4-aac0-4f03-97af-5289530b3ca9","Type":"ContainerStarted","Data":"150914f3609927829aeb425a0ebdb10dfac66dbf228e2137ecc7941a868dc0b3"} Jul 08 01:52:25.390941 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.390844 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" podUID="9ea16ad4-aac0-4f03-97af-5289530b3ca9" containerName="authorino" containerID="cri-o://370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806" gracePeriod=30 Jul 08 01:52:25.392367 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.392338 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66d74db596-b8ndd" event={"ID":"4e83845f-1b24-409e-8ba5-d7eb9a4b8b29","Type":"ContainerStarted","Data":"927f5cc21582c0f7d1f04653aae122cff2e7696f94d31c6f39ef6412ec48b203"} Jul 08 01:52:25.405966 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.405929 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" podStartSLOduration=1.121589577 podStartE2EDuration="1.405916059s" totalImagesPullingTime="284.326482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:52:25.405089027 +0000 UTC m=+576.067003564" watchObservedRunningTime="2026-07-08 01:52:25.405916059 +0000 UTC m=+576.067830595" Jul 08 01:52:25.415440 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.415418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb21ed03-0acd-46bb-aaae-da39d4c73972-tls-cert\") pod \"authorino-7957c9d8b8-fxc9l\" (UID: \"fb21ed03-0acd-46bb-aaae-da39d4c73972\") " pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:52:25.415543 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.415499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw9zt\" (UniqueName: \"kubernetes.io/projected/fb21ed03-0acd-46bb-aaae-da39d4c73972-kube-api-access-jw9zt\") pod \"authorino-7957c9d8b8-fxc9l\" (UID: \"fb21ed03-0acd-46bb-aaae-da39d4c73972\") " pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:52:25.516918 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.516889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb21ed03-0acd-46bb-aaae-da39d4c73972-tls-cert\") pod \"authorino-7957c9d8b8-fxc9l\" (UID: \"fb21ed03-0acd-46bb-aaae-da39d4c73972\") " pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:52:25.517026 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.516980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw9zt\" (UniqueName: \"kubernetes.io/projected/fb21ed03-0acd-46bb-aaae-da39d4c73972-kube-api-access-jw9zt\") pod \"authorino-7957c9d8b8-fxc9l\" (UID: \"fb21ed03-0acd-46bb-aaae-da39d4c73972\") " pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:52:25.519142 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.519124 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb21ed03-0acd-46bb-aaae-da39d4c73972-tls-cert\") pod \"authorino-7957c9d8b8-fxc9l\" (UID: \"fb21ed03-0acd-46bb-aaae-da39d4c73972\") " pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:52:25.524021 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.523998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw9zt\" (UniqueName: \"kubernetes.io/projected/fb21ed03-0acd-46bb-aaae-da39d4c73972-kube-api-access-jw9zt\") pod \"authorino-7957c9d8b8-fxc9l\" (UID: \"fb21ed03-0acd-46bb-aaae-da39d4c73972\") " pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:52:25.593962 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.593929 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:52:25.621865 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.621838 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" Jul 08 01:52:25.717904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.717880 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcm88\" (UniqueName: \"kubernetes.io/projected/9ea16ad4-aac0-4f03-97af-5289530b3ca9-kube-api-access-vcm88\") pod \"9ea16ad4-aac0-4f03-97af-5289530b3ca9\" (UID: \"9ea16ad4-aac0-4f03-97af-5289530b3ca9\") " Jul 08 01:52:25.720429 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.720403 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea16ad4-aac0-4f03-97af-5289530b3ca9-kube-api-access-vcm88" (OuterVolumeSpecName: "kube-api-access-vcm88") pod "9ea16ad4-aac0-4f03-97af-5289530b3ca9" (UID: "9ea16ad4-aac0-4f03-97af-5289530b3ca9"). InnerVolumeSpecName "kube-api-access-vcm88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:52:25.726216 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.726195 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7957c9d8b8-fxc9l"] Jul 08 01:52:25.728535 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:52:25.728513 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb21ed03_0acd_46bb_aaae_da39d4c73972.slice/crio-1d6c4285099b97eefecdeffc19fd47004a15eaf4a180f8f61fabf5d884591ccd WatchSource:0}: Error finding container 1d6c4285099b97eefecdeffc19fd47004a15eaf4a180f8f61fabf5d884591ccd: Status 404 returned error can't find the container with id 1d6c4285099b97eefecdeffc19fd47004a15eaf4a180f8f61fabf5d884591ccd Jul 08 01:52:25.819286 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:25.819209 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcm88\" (UniqueName: \"kubernetes.io/projected/9ea16ad4-aac0-4f03-97af-5289530b3ca9-kube-api-access-vcm88\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:52:26.398798 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.398761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66d74db596-b8ndd" event={"ID":"4e83845f-1b24-409e-8ba5-d7eb9a4b8b29","Type":"ContainerStarted","Data":"4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d"} Jul 08 01:52:26.398986 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.398811 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-66d74db596-b8ndd" podUID="4e83845f-1b24-409e-8ba5-d7eb9a4b8b29" containerName="authorino" containerID="cri-o://4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d" gracePeriod=30 Jul 08 01:52:26.400088 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.400062 2578 generic.go:356] "Generic (PLEG): container finished" podID="9ea16ad4-aac0-4f03-97af-5289530b3ca9" containerID="370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806" exitCode=2 Jul 08 01:52:26.400217 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.400093 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" Jul 08 01:52:26.400217 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.400153 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" event={"ID":"9ea16ad4-aac0-4f03-97af-5289530b3ca9","Type":"ContainerDied","Data":"370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806"} Jul 08 01:52:26.400217 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.400206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-vwb5m" event={"ID":"9ea16ad4-aac0-4f03-97af-5289530b3ca9","Type":"ContainerDied","Data":"150914f3609927829aeb425a0ebdb10dfac66dbf228e2137ecc7941a868dc0b3"} Jul 08 01:52:26.400384 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.400232 2578 scope.go:117] "RemoveContainer" containerID="370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806" Jul 08 01:52:26.401824 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.401797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" event={"ID":"fb21ed03-0acd-46bb-aaae-da39d4c73972","Type":"ContainerStarted","Data":"e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659"} Jul 08 01:52:26.401941 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.401837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" event={"ID":"fb21ed03-0acd-46bb-aaae-da39d4c73972","Type":"ContainerStarted","Data":"1d6c4285099b97eefecdeffc19fd47004a15eaf4a180f8f61fabf5d884591ccd"} Jul 08 01:52:26.409236 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.409220 2578 scope.go:117] "RemoveContainer" containerID="370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806" Jul 08 01:52:26.409520 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:52:26.409503 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806\": container with ID starting with 370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806 not found: ID does not exist" containerID="370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806" Jul 08 01:52:26.409598 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.409528 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806"} err="failed to get container status \"370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806\": rpc error: code = NotFound desc = could not find container \"370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806\": container with ID starting with 370d47ed02b928ff4f00e3dbc5cb0565b7cc3d787cf60d596acaa1087fe4f806 not found: ID does not exist" Jul 08 01:52:26.415817 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.415777 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66d74db596-b8ndd" podStartSLOduration=2.116581886 podStartE2EDuration="2.415764417s" totalImagesPullingTime="299.182531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:52:26.414465099 +0000 UTC m=+577.076379656" watchObservedRunningTime="2026-07-08 01:52:26.415764417 +0000 UTC m=+577.077678953" Jul 08 01:52:26.429313 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.429276 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" podStartSLOduration=1.140282253 podStartE2EDuration="1.429263003s" totalImagesPullingTime="288.98075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:52:26.429042857 +0000 UTC m=+577.090957406" watchObservedRunningTime="2026-07-08 01:52:26.429263003 +0000 UTC m=+577.091177541" Jul 08 01:52:26.451070 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.451045 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vwb5m"] Jul 08 01:52:26.458361 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.458329 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vwb5m"] Jul 08 01:52:26.461704 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.461676 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t8scl"] Jul 08 01:52:26.461854 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.461829 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-t8scl" podUID="4d3eb237-590b-4b8c-8f22-794491364809" containerName="authorino" containerID="cri-o://3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216" gracePeriod=30 Jul 08 01:52:26.692259 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.692232 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66d74db596-b8ndd" Jul 08 01:52:26.728769 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.728741 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t8scl" Jul 08 01:52:26.827358 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.827332 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwk8n\" (UniqueName: \"kubernetes.io/projected/4d3eb237-590b-4b8c-8f22-794491364809-kube-api-access-cwk8n\") pod \"4d3eb237-590b-4b8c-8f22-794491364809\" (UID: \"4d3eb237-590b-4b8c-8f22-794491364809\") " Jul 08 01:52:26.827516 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.827398 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkd48\" (UniqueName: \"kubernetes.io/projected/4e83845f-1b24-409e-8ba5-d7eb9a4b8b29-kube-api-access-fkd48\") pod \"4e83845f-1b24-409e-8ba5-d7eb9a4b8b29\" (UID: \"4e83845f-1b24-409e-8ba5-d7eb9a4b8b29\") " Jul 08 01:52:26.829451 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.829428 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3eb237-590b-4b8c-8f22-794491364809-kube-api-access-cwk8n" (OuterVolumeSpecName: "kube-api-access-cwk8n") pod "4d3eb237-590b-4b8c-8f22-794491364809" (UID: "4d3eb237-590b-4b8c-8f22-794491364809"). InnerVolumeSpecName "kube-api-access-cwk8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:52:26.829554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.829468 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e83845f-1b24-409e-8ba5-d7eb9a4b8b29-kube-api-access-fkd48" (OuterVolumeSpecName: "kube-api-access-fkd48") pod "4e83845f-1b24-409e-8ba5-d7eb9a4b8b29" (UID: "4e83845f-1b24-409e-8ba5-d7eb9a4b8b29"). InnerVolumeSpecName "kube-api-access-fkd48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:52:26.928773 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.928708 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkd48\" (UniqueName: \"kubernetes.io/projected/4e83845f-1b24-409e-8ba5-d7eb9a4b8b29-kube-api-access-fkd48\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:52:26.928773 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:26.928733 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwk8n\" (UniqueName: \"kubernetes.io/projected/4d3eb237-590b-4b8c-8f22-794491364809-kube-api-access-cwk8n\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:52:27.407159 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.407125 2578 generic.go:356] "Generic (PLEG): container finished" podID="4e83845f-1b24-409e-8ba5-d7eb9a4b8b29" containerID="4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d" exitCode=0 Jul 08 01:52:27.407294 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.407164 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66d74db596-b8ndd" Jul 08 01:52:27.407294 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.407197 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66d74db596-b8ndd" event={"ID":"4e83845f-1b24-409e-8ba5-d7eb9a4b8b29","Type":"ContainerDied","Data":"4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d"} Jul 08 01:52:27.407294 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.407228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66d74db596-b8ndd" event={"ID":"4e83845f-1b24-409e-8ba5-d7eb9a4b8b29","Type":"ContainerDied","Data":"927f5cc21582c0f7d1f04653aae122cff2e7696f94d31c6f39ef6412ec48b203"} Jul 08 01:52:27.407294 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.407244 2578 scope.go:117] "RemoveContainer" containerID="4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d" Jul 08 01:52:27.409424 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.409401 2578 generic.go:356] "Generic (PLEG): container finished" podID="4d3eb237-590b-4b8c-8f22-794491364809" containerID="3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216" exitCode=0 Jul 08 01:52:27.409554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.409425 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t8scl" event={"ID":"4d3eb237-590b-4b8c-8f22-794491364809","Type":"ContainerDied","Data":"3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216"} Jul 08 01:52:27.409554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.409443 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t8scl" Jul 08 01:52:27.409554 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.409459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t8scl" event={"ID":"4d3eb237-590b-4b8c-8f22-794491364809","Type":"ContainerDied","Data":"6a06742f557dd02f67e80c493003bcff5ade0468258dfcc0d8c7936ec15f3477"} Jul 08 01:52:27.417159 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.417141 2578 scope.go:117] "RemoveContainer" containerID="4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d" Jul 08 01:52:27.417382 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:52:27.417364 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d\": container with ID starting with 4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d not found: ID does not exist" containerID="4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d" Jul 08 01:52:27.417460 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.417388 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d"} err="failed to get container status \"4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d\": rpc error: code = NotFound desc = could not find container \"4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d\": container with ID starting with 4cf273d801fb021bcb48cc082fd64adf8a8b3ddca556b0ab3f9b7fb0e3c5065d not found: ID does not exist" Jul 08 01:52:27.417460 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.417405 2578 scope.go:117] "RemoveContainer" containerID="3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216" Jul 08 01:52:27.424849 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.424832 2578 scope.go:117] "RemoveContainer" containerID="3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216" Jul 08 01:52:27.425074 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:52:27.425056 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216\": container with ID starting with 3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216 not found: ID does not exist" containerID="3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216" Jul 08 01:52:27.425117 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.425080 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216"} err="failed to get container status \"3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216\": rpc error: code = NotFound desc = could not find container \"3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216\": container with ID starting with 3367c8233dba166018c7cadc7750cecb75f090e5be518ecea3ea27914f2ea216 not found: ID does not exist" Jul 08 01:52:27.432705 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.432685 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66d74db596-b8ndd"] Jul 08 01:52:27.436975 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.436957 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66d74db596-b8ndd"] Jul 08 01:52:27.447787 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.447763 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t8scl"] Jul 08 01:52:27.451633 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.451597 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t8scl"] Jul 08 01:52:27.905877 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.905843 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d3eb237-590b-4b8c-8f22-794491364809" path="/var/lib/kubelet/pods/4d3eb237-590b-4b8c-8f22-794491364809/volumes" Jul 08 01:52:27.906311 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.906278 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e83845f-1b24-409e-8ba5-d7eb9a4b8b29" path="/var/lib/kubelet/pods/4e83845f-1b24-409e-8ba5-d7eb9a4b8b29/volumes" Jul 08 01:52:27.906622 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:52:27.906604 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea16ad4-aac0-4f03-97af-5289530b3ca9" path="/var/lib/kubelet/pods/9ea16ad4-aac0-4f03-97af-5289530b3ca9/volumes" Jul 08 01:53:08.088646 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:53:08.088614 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:53:27.317969 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:53:27.317929 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:53:37.525115 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:53:37.525079 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:53:43.612646 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:53:43.612614 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:53:51.117911 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:53:51.117870 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:53:58.613926 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:53:58.613891 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:54:02.536889 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.536853 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj"] Jul 08 01:54:02.537243 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.537174 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea16ad4-aac0-4f03-97af-5289530b3ca9" containerName="authorino" Jul 08 01:54:02.537243 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.537186 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea16ad4-aac0-4f03-97af-5289530b3ca9" containerName="authorino" Jul 08 01:54:02.537243 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.537204 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e83845f-1b24-409e-8ba5-d7eb9a4b8b29" containerName="authorino" Jul 08 01:54:02.537243 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.537209 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e83845f-1b24-409e-8ba5-d7eb9a4b8b29" containerName="authorino" Jul 08 01:54:02.537243 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.537217 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d3eb237-590b-4b8c-8f22-794491364809" containerName="authorino" Jul 08 01:54:02.537243 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.537223 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d3eb237-590b-4b8c-8f22-794491364809" containerName="authorino" Jul 08 01:54:02.537465 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.537280 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e83845f-1b24-409e-8ba5-d7eb9a4b8b29" containerName="authorino" Jul 08 01:54:02.537465 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.537289 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea16ad4-aac0-4f03-97af-5289530b3ca9" containerName="authorino" Jul 08 01:54:02.537465 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.537296 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d3eb237-590b-4b8c-8f22-794491364809" containerName="authorino" Jul 08 01:54:02.540369 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.540354 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.543377 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.543360 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 01:54:02.544406 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.544372 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hz9bp\"" Jul 08 01:54:02.544552 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.544417 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 08 01:54:02.544552 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.544432 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 01:54:02.550859 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.550839 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj"] Jul 08 01:54:02.620621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.620587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.620621 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.620626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.620856 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.620652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtbhd\" (UniqueName: \"kubernetes.io/projected/40369d2a-3ead-4274-9f8d-e125abd67eaf-kube-api-access-qtbhd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.620856 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.620678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40369d2a-3ead-4274-9f8d-e125abd67eaf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.620856 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.620736 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.620856 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.620791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.721735 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.721700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.721904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.721780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.721904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.721807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.721904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.721835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtbhd\" (UniqueName: \"kubernetes.io/projected/40369d2a-3ead-4274-9f8d-e125abd67eaf-kube-api-access-qtbhd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.721904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.721863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40369d2a-3ead-4274-9f8d-e125abd67eaf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.721904 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.721894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.722177 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.722154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.722240 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.722184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.722240 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.722218 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.724114 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.724090 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40369d2a-3ead-4274-9f8d-e125abd67eaf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.724354 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.724337 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40369d2a-3ead-4274-9f8d-e125abd67eaf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.729604 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.729584 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtbhd\" (UniqueName: \"kubernetes.io/projected/40369d2a-3ead-4274-9f8d-e125abd67eaf-kube-api-access-qtbhd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj\" (UID: \"40369d2a-3ead-4274-9f8d-e125abd67eaf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.850335 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.850311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:02.980061 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.980034 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj"] Jul 08 01:54:02.982412 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:54:02.982374 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40369d2a_3ead_4274_9f8d_e125abd67eaf.slice/crio-c4f64a586360778b4ce866ab146b8c4944813895ec571750477078e96447890c WatchSource:0}: Error finding container c4f64a586360778b4ce866ab146b8c4944813895ec571750477078e96447890c: Status 404 returned error can't find the container with id c4f64a586360778b4ce866ab146b8c4944813895ec571750477078e96447890c Jul 08 01:54:02.983675 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:02.983656 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 01:54:03.840016 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:03.839975 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" event={"ID":"40369d2a-3ead-4274-9f8d-e125abd67eaf","Type":"ContainerStarted","Data":"c4f64a586360778b4ce866ab146b8c4944813895ec571750477078e96447890c"} Jul 08 01:54:04.845845 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:04.845807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" event={"ID":"40369d2a-3ead-4274-9f8d-e125abd67eaf","Type":"ContainerStarted","Data":"e991a8915ee088bc1779c3d3268b35224bb0624f63a005f498c0a2b1ea28a683"} Jul 08 01:54:04.864262 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:04.864212 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" podStartSLOduration=1.863190833 podStartE2EDuration="2.864194122s" totalImagesPullingTime="1.001003289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:54:04.862645744 +0000 UTC m=+675.524560281" watchObservedRunningTime="2026-07-08 01:54:04.864194122 +0000 UTC m=+675.526108659" Jul 08 01:54:05.510389 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:05.510353 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:54:12.851060 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:12.851030 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:12.851543 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:12.851072 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:12.863289 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:12.863264 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:12.890182 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:12.890165 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj" Jul 08 01:54:43.708051 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:43.707978 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-575bbf8489-jq5t7"] Jul 08 01:54:43.711354 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:43.711337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575bbf8489-jq5t7" Jul 08 01:54:43.721046 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:43.721024 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-575bbf8489-jq5t7"] Jul 08 01:54:43.783974 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:43.783946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e311fc41-f7c2-4f54-8dee-c577ee2a6942-tls-cert\") pod \"authorino-575bbf8489-jq5t7\" (UID: \"e311fc41-f7c2-4f54-8dee-c577ee2a6942\") " pod="kuadrant-system/authorino-575bbf8489-jq5t7" Jul 08 01:54:43.783974 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:43.783976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6shf\" (UniqueName: \"kubernetes.io/projected/e311fc41-f7c2-4f54-8dee-c577ee2a6942-kube-api-access-z6shf\") pod \"authorino-575bbf8489-jq5t7\" (UID: \"e311fc41-f7c2-4f54-8dee-c577ee2a6942\") " pod="kuadrant-system/authorino-575bbf8489-jq5t7" Jul 08 01:54:43.884950 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:43.884919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e311fc41-f7c2-4f54-8dee-c577ee2a6942-tls-cert\") pod \"authorino-575bbf8489-jq5t7\" (UID: \"e311fc41-f7c2-4f54-8dee-c577ee2a6942\") " pod="kuadrant-system/authorino-575bbf8489-jq5t7" Jul 08 01:54:43.885077 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:43.884968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6shf\" (UniqueName: \"kubernetes.io/projected/e311fc41-f7c2-4f54-8dee-c577ee2a6942-kube-api-access-z6shf\") pod \"authorino-575bbf8489-jq5t7\" (UID: \"e311fc41-f7c2-4f54-8dee-c577ee2a6942\") " pod="kuadrant-system/authorino-575bbf8489-jq5t7" Jul 08 01:54:43.887273 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:43.887253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e311fc41-f7c2-4f54-8dee-c577ee2a6942-tls-cert\") pod \"authorino-575bbf8489-jq5t7\" (UID: \"e311fc41-f7c2-4f54-8dee-c577ee2a6942\") " pod="kuadrant-system/authorino-575bbf8489-jq5t7" Jul 08 01:54:43.893000 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:43.892981 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6shf\" (UniqueName: \"kubernetes.io/projected/e311fc41-f7c2-4f54-8dee-c577ee2a6942-kube-api-access-z6shf\") pod \"authorino-575bbf8489-jq5t7\" (UID: \"e311fc41-f7c2-4f54-8dee-c577ee2a6942\") " pod="kuadrant-system/authorino-575bbf8489-jq5t7" Jul 08 01:54:44.020358 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:44.020293 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575bbf8489-jq5t7" Jul 08 01:54:44.149825 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:44.149795 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-575bbf8489-jq5t7"] Jul 08 01:54:44.153231 ip-10-0-134-84 kubenswrapper[2578]: W0708 01:54:44.153201 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode311fc41_f7c2_4f54_8dee_c577ee2a6942.slice/crio-3f8c190d71adcb7c9beb7096cba9e69618a9b27ff13d3f803282320bd7ed113a WatchSource:0}: Error finding container 3f8c190d71adcb7c9beb7096cba9e69618a9b27ff13d3f803282320bd7ed113a: Status 404 returned error can't find the container with id 3f8c190d71adcb7c9beb7096cba9e69618a9b27ff13d3f803282320bd7ed113a Jul 08 01:54:45.023807 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.023756 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575bbf8489-jq5t7" event={"ID":"e311fc41-f7c2-4f54-8dee-c577ee2a6942","Type":"ContainerStarted","Data":"2a34536ac75ded888f993d019bad058bb80c05c822de17c2d248557b605dc9ad"} Jul 08 01:54:45.024351 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.023814 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575bbf8489-jq5t7" event={"ID":"e311fc41-f7c2-4f54-8dee-c577ee2a6942","Type":"ContainerStarted","Data":"3f8c190d71adcb7c9beb7096cba9e69618a9b27ff13d3f803282320bd7ed113a"} Jul 08 01:54:45.043942 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.043877 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-575bbf8489-jq5t7" podStartSLOduration=1.72679402 podStartE2EDuration="2.043856347s" totalImagesPullingTime="317.062327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 01:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 01:54:45.04215704 +0000 UTC m=+715.704071578" watchObservedRunningTime="2026-07-08 01:54:45.043856347 +0000 UTC m=+715.705770885" Jul 08 01:54:45.077557 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.077441 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7957c9d8b8-fxc9l"] Jul 08 01:54:45.077739 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.077675 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" podUID="fb21ed03-0acd-46bb-aaae-da39d4c73972" containerName="authorino" containerID="cri-o://e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659" gracePeriod=30 Jul 08 01:54:45.350234 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.350208 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:54:45.397181 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.397147 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw9zt\" (UniqueName: \"kubernetes.io/projected/fb21ed03-0acd-46bb-aaae-da39d4c73972-kube-api-access-jw9zt\") pod \"fb21ed03-0acd-46bb-aaae-da39d4c73972\" (UID: \"fb21ed03-0acd-46bb-aaae-da39d4c73972\") " Jul 08 01:54:45.397329 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.397218 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb21ed03-0acd-46bb-aaae-da39d4c73972-tls-cert\") pod \"fb21ed03-0acd-46bb-aaae-da39d4c73972\" (UID: \"fb21ed03-0acd-46bb-aaae-da39d4c73972\") " Jul 08 01:54:45.399168 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.399133 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb21ed03-0acd-46bb-aaae-da39d4c73972-kube-api-access-jw9zt" (OuterVolumeSpecName: "kube-api-access-jw9zt") pod "fb21ed03-0acd-46bb-aaae-da39d4c73972" (UID: "fb21ed03-0acd-46bb-aaae-da39d4c73972"). InnerVolumeSpecName "kube-api-access-jw9zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 01:54:45.407457 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.407421 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb21ed03-0acd-46bb-aaae-da39d4c73972-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "fb21ed03-0acd-46bb-aaae-da39d4c73972" (UID: "fb21ed03-0acd-46bb-aaae-da39d4c73972"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 01:54:45.498097 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.498063 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw9zt\" (UniqueName: \"kubernetes.io/projected/fb21ed03-0acd-46bb-aaae-da39d4c73972-kube-api-access-jw9zt\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:54:45.498097 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:45.498089 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb21ed03-0acd-46bb-aaae-da39d4c73972-tls-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 01:54:46.030514 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:46.030460 2578 generic.go:356] "Generic (PLEG): container finished" podID="fb21ed03-0acd-46bb-aaae-da39d4c73972" containerID="e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659" exitCode=0 Jul 08 01:54:46.030927 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:46.030521 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" Jul 08 01:54:46.030927 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:46.030551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" event={"ID":"fb21ed03-0acd-46bb-aaae-da39d4c73972","Type":"ContainerDied","Data":"e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659"} Jul 08 01:54:46.030927 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:46.030601 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7957c9d8b8-fxc9l" event={"ID":"fb21ed03-0acd-46bb-aaae-da39d4c73972","Type":"ContainerDied","Data":"1d6c4285099b97eefecdeffc19fd47004a15eaf4a180f8f61fabf5d884591ccd"} Jul 08 01:54:46.030927 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:46.030626 2578 scope.go:117] "RemoveContainer" containerID="e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659" Jul 08 01:54:46.039334 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:46.039310 2578 scope.go:117] "RemoveContainer" containerID="e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659" Jul 08 01:54:46.039665 ip-10-0-134-84 kubenswrapper[2578]: E0708 01:54:46.039635 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659\": container with ID starting with e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659 not found: ID does not exist" containerID="e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659" Jul 08 01:54:46.039773 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:46.039674 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659"} err="failed to get container status \"e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659\": rpc error: code = NotFound desc = could not find container \"e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659\": container with ID starting with e69d33a8ba0affa3415a2614b341d2337935155d759745580b858c05e9b38659 not found: ID does not exist" Jul 08 01:54:46.049942 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:46.049911 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7957c9d8b8-fxc9l"] Jul 08 01:54:46.062104 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:46.062079 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7957c9d8b8-fxc9l"] Jul 08 01:54:47.906090 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:54:47.906058 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb21ed03-0acd-46bb-aaae-da39d4c73972" path="/var/lib/kubelet/pods/fb21ed03-0acd-46bb-aaae-da39d4c73972/volumes" Jul 08 01:55:14.613939 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:55:14.613900 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:55:25.012233 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:55:25.012198 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:55:35.835529 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:55:35.835494 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:55:45.016422 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:55:45.016388 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:55:56.110108 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:55:56.110073 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:56:23.717925 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:56:23.717886 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:56:34.315025 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:56:34.314989 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:57:10.714942 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:57:10.714909 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:57:18.513836 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:57:18.513800 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:57:29.210092 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:57:29.210055 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:57:36.911119 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:57:36.911085 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:57:46.900286 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:57:46.900204 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:57:58.708669 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:57:58.708632 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:58:09.216034 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:58:09.216000 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:58:19.007528 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:58:19.007500 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:58:41.912243 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:58:41.912209 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:58:51.204048 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:58:51.204015 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:59:13.106193 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:59:13.106114 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:59:23.712599 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:59:23.712564 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:59:40.315164 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:59:40.315132 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 01:59:50.121349 ip-10-0-134-84 kubenswrapper[2578]: I0708 01:59:50.121310 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:00:00.812660 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:00:00.812613 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:00:10.410170 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:00:10.410134 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:00:19.813084 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:00:19.813051 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:02:09.617865 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:02:09.617831 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:02:18.612742 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:02:18.612667 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:02:31.711443 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:02:31.711410 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:02:43.609362 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:02:43.609329 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:02:55.815753 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:02:55.815716 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:03:05.812352 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:03:05.812315 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:03:13.743604 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:03:13.743574 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:03:25.808495 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:03:25.808439 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:03:35.212531 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:03:35.212496 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:03:45.512214 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:03:45.512176 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:03:59.014750 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:03:59.014713 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:04:17.220419 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:04:17.220388 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:04:26.312787 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:04:26.312748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:04:36.814790 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:04:36.814752 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:04:46.505334 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:04:46.505300 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:05:03.311493 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:05:03.311444 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:05:12.808537 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:05:12.808444 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:05:21.817786 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:05:21.817755 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:05:31.933362 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:05:31.933327 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:05:43.107332 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:05:43.107293 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:05:52.610139 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:05:52.610102 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:06:19.806368 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:06:19.806336 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:06:29.601961 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:06:29.601922 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:06:39.508660 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:06:39.508626 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:06:48.507865 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:06:48.507789 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:07:00.903831 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:07:00.903798 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:07:12.002808 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:07:12.002773 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:07:20.909344 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:07:20.909310 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:07:33.505115 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:07:33.505071 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:07:45.412434 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:07:45.412376 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:07:59.099234 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:07:59.099194 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:08:12.913720 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:08:12.913641 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:08:22.200838 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:08:22.200787 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:08:31.201879 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:08:31.201847 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:08:51.206509 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:08:51.206461 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:09:05.414600 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:09:05.414567 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:09:21.409878 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:09:21.409839 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:09:30.503423 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:09:30.503385 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:11:14.706601 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:11:14.706525 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:11:22.827913 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:11:22.827877 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:12:44.915996 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:12:44.915904 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:12:55.014804 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:12:55.014774 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:13:04.704639 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:04.704605 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:13:31.510397 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:31.510361 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:13:43.650852 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.650818 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt"] Jul 08 02:13:43.651225 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.651171 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb21ed03-0acd-46bb-aaae-da39d4c73972" containerName="authorino" Jul 08 02:13:43.651225 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.651182 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb21ed03-0acd-46bb-aaae-da39d4c73972" containerName="authorino" Jul 08 02:13:43.651301 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.651237 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb21ed03-0acd-46bb-aaae-da39d4c73972" containerName="authorino" Jul 08 02:13:43.654441 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.654422 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.657584 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.657153 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b64edcbf\"/\"openshift-service-ca.crt\"" Jul 08 02:13:43.657584 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.657211 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b64edcbf\"/\"kube-root-ca.crt\"" Jul 08 02:13:43.658101 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.658079 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b64edcbf\"/\"e2e-lifecycle-model-b64edcbf-kserve-self-signed-certs\"" Jul 08 02:13:43.658211 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.658111 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b64edcbf\"/\"default-dockercfg-fnl27\"" Jul 08 02:13:43.664617 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.664594 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt"] Jul 08 02:13:43.794954 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.794923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zvb\" (UniqueName: \"kubernetes.io/projected/f940c805-d1fa-4fdc-85f3-138e68d5ec45-kube-api-access-f2zvb\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.794954 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.794957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-dshm\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.795142 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.794975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-model-cache\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.795142 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.795029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tls-certs\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.795142 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.795077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-home\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.795142 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.795100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tmp-dir\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.895972 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.895923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zvb\" (UniqueName: \"kubernetes.io/projected/f940c805-d1fa-4fdc-85f3-138e68d5ec45-kube-api-access-f2zvb\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.895972 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.895958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-dshm\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.896153 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.895975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-model-cache\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.896153 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.896006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tls-certs\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.896251 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.896184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-home\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.896251 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.896210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tmp-dir\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.896491 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.896452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-model-cache\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.896588 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.896555 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-home\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.896641 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.896605 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tmp-dir\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.899182 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.899160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tls-certs\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.899386 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.899369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-dshm\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.905426 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.905373 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zvb\" (UniqueName: \"kubernetes.io/projected/f940c805-d1fa-4fdc-85f3-138e68d5ec45-kube-api-access-f2zvb\") pod \"e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:43.966024 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:43.966002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:44.095625 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:44.095594 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt"] Jul 08 02:13:44.099158 ip-10-0-134-84 kubenswrapper[2578]: W0708 02:13:44.099132 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf940c805_d1fa_4fdc_85f3_138e68d5ec45.slice/crio-91d4b26cb0f1e4b4ec53b35be99bb322cbf43b0e8042ef674c42d4f51d57c2fd WatchSource:0}: Error finding container 91d4b26cb0f1e4b4ec53b35be99bb322cbf43b0e8042ef674c42d4f51d57c2fd: Status 404 returned error can't find the container with id 91d4b26cb0f1e4b4ec53b35be99bb322cbf43b0e8042ef674c42d4f51d57c2fd Jul 08 02:13:45.049117 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:45.049080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" event={"ID":"f940c805-d1fa-4fdc-85f3-138e68d5ec45","Type":"ContainerStarted","Data":"bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911"} Jul 08 02:13:45.049117 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:45.049125 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" event={"ID":"f940c805-d1fa-4fdc-85f3-138e68d5ec45","Type":"ContainerStarted","Data":"91d4b26cb0f1e4b4ec53b35be99bb322cbf43b0e8042ef674c42d4f51d57c2fd"} Jul 08 02:13:45.065262 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:45.065216 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" podStartSLOduration=2.065202093 podStartE2EDuration="2.065202093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:13:45.063895195 +0000 UTC m=+1855.725809733" watchObservedRunningTime="2026-07-08 02:13:45.065202093 +0000 UTC m=+1855.727116629" Jul 08 02:13:47.405086 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:47.405049 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:13:53.966979 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:53.966945 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:53.966979 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:53.966983 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:53.979041 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:53.979011 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:13:54.099851 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:13:54.099825 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:14:05.114161 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:05.114127 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:14:18.769537 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:18.769445 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt"] Jul 08 02:14:20.506892 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:20.506853 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:14:33.789667 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:33.789608 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" podUID="f940c805-d1fa-4fdc-85f3-138e68d5ec45" containerName="main" containerID="cri-o://bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911" gracePeriod=45 Jul 08 02:14:34.037555 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.037531 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:14:34.133230 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133192 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-model-cache\") pod \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " Jul 08 02:14:34.133407 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133312 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2zvb\" (UniqueName: \"kubernetes.io/projected/f940c805-d1fa-4fdc-85f3-138e68d5ec45-kube-api-access-f2zvb\") pod \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " Jul 08 02:14:34.133407 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133347 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-home\") pod \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " Jul 08 02:14:34.133407 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133371 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-dshm\") pod \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " Jul 08 02:14:34.133407 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133396 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tls-certs\") pod \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " Jul 08 02:14:34.133590 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133429 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tmp-dir\") pod \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\" (UID: \"f940c805-d1fa-4fdc-85f3-138e68d5ec45\") " Jul 08 02:14:34.133590 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133464 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-model-cache" (OuterVolumeSpecName: "model-cache") pod "f940c805-d1fa-4fdc-85f3-138e68d5ec45" (UID: "f940c805-d1fa-4fdc-85f3-138e68d5ec45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:14:34.133721 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133649 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-home" (OuterVolumeSpecName: "home") pod "f940c805-d1fa-4fdc-85f3-138e68d5ec45" (UID: "f940c805-d1fa-4fdc-85f3-138e68d5ec45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:14:34.133797 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133768 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f940c805-d1fa-4fdc-85f3-138e68d5ec45" (UID: "f940c805-d1fa-4fdc-85f3-138e68d5ec45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:14:34.133856 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133782 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:14:34.133856 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.133825 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:14:34.135576 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.135546 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-dshm" (OuterVolumeSpecName: "dshm") pod "f940c805-d1fa-4fdc-85f3-138e68d5ec45" (UID: "f940c805-d1fa-4fdc-85f3-138e68d5ec45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:14:34.135663 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.135627 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f940c805-d1fa-4fdc-85f3-138e68d5ec45-kube-api-access-f2zvb" (OuterVolumeSpecName: "kube-api-access-f2zvb") pod "f940c805-d1fa-4fdc-85f3-138e68d5ec45" (UID: "f940c805-d1fa-4fdc-85f3-138e68d5ec45"). InnerVolumeSpecName "kube-api-access-f2zvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:14:34.135705 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.135657 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f940c805-d1fa-4fdc-85f3-138e68d5ec45" (UID: "f940c805-d1fa-4fdc-85f3-138e68d5ec45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:14:34.234520 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.234463 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zvb\" (UniqueName: \"kubernetes.io/projected/f940c805-d1fa-4fdc-85f3-138e68d5ec45-kube-api-access-f2zvb\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:14:34.234520 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.234517 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:14:34.234520 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.234527 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:14:34.234716 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.234537 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f940c805-d1fa-4fdc-85f3-138e68d5ec45-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:14:34.267363 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.267338 2578 generic.go:356] "Generic (PLEG): container finished" podID="f940c805-d1fa-4fdc-85f3-138e68d5ec45" containerID="bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911" exitCode=0 Jul 08 02:14:34.267472 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.267380 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" Jul 08 02:14:34.267472 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.267424 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" event={"ID":"f940c805-d1fa-4fdc-85f3-138e68d5ec45","Type":"ContainerDied","Data":"bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911"} Jul 08 02:14:34.267472 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.267465 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt" event={"ID":"f940c805-d1fa-4fdc-85f3-138e68d5ec45","Type":"ContainerDied","Data":"91d4b26cb0f1e4b4ec53b35be99bb322cbf43b0e8042ef674c42d4f51d57c2fd"} Jul 08 02:14:34.267632 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.267496 2578 scope.go:117] "RemoveContainer" containerID="bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911" Jul 08 02:14:34.276256 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.276236 2578 scope.go:117] "RemoveContainer" containerID="bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911" Jul 08 02:14:34.276555 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:14:34.276466 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911\": container with ID starting with bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911 not found: ID does not exist" containerID="bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911" Jul 08 02:14:34.276606 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.276569 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911"} err="failed to get container status \"bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911\": rpc error: code = NotFound desc = could not find container \"bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911\": container with ID starting with bc8dd6bfd5af7322a434615db6dbe8138b76a880c06dc5aa505fc3433b9df911 not found: ID does not exist" Jul 08 02:14:34.288031 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.288006 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt"] Jul 08 02:14:34.291909 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:34.291891 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b64edcbf/e2e-lifecycle-model-b64edcbf-kserve-65c555d875-p2bwt"] Jul 08 02:14:35.906323 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:35.906278 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f940c805-d1fa-4fdc-85f3-138e68d5ec45" path="/var/lib/kubelet/pods/f940c805-d1fa-4fdc-85f3-138e68d5ec45/volumes" Jul 08 02:14:58.507058 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:14:58.507025 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:15:12.107811 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:15:12.107780 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:15:22.007951 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:15:22.007919 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:16:31.803718 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:16:31.803684 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:17:15.923384 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:15.923307 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl"] Jul 08 02:17:15.923841 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:15.923661 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f940c805-d1fa-4fdc-85f3-138e68d5ec45" containerName="main" Jul 08 02:17:15.923841 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:15.923673 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f940c805-d1fa-4fdc-85f3-138e68d5ec45" containerName="main" Jul 08 02:17:15.923841 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:15.923736 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f940c805-d1fa-4fdc-85f3-138e68d5ec45" containerName="main" Jul 08 02:17:15.926597 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:15.926579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:15.936719 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:15.936698 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-219487-openshift-default-dockercfg-xxmlz\"" Jul 08 02:17:15.959226 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:15.959199 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl"] Jul 08 02:17:16.030025 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.029993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-envoy\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.030172 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.030031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-credential-socket\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.030172 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.030056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-socket\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.030172 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.030110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-token\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.030172 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.030163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istiod-ca-cert\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.030308 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.030181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwbv\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-kube-api-access-ltwbv\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.030308 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.030247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-certs\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.030308 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.030305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-podinfo\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.030403 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.030332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-data\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131238 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-credential-socket\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-socket\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-token\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istiod-ca-cert\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwbv\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-kube-api-access-ltwbv\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-certs\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131690 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-podinfo\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131690 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-data\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131690 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-envoy\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131857 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131787 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-socket\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131857 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131798 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-certs\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131966 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131908 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-data\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.131966 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.131914 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-credential-socket\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.132064 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.132037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istiod-ca-cert\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.133768 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.133747 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-envoy\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.134001 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.133984 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-podinfo\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.139799 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.139773 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-token\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.139875 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.139834 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwbv\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-kube-api-access-ltwbv\") pod \"e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.236113 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.236047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:16.369160 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.369133 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl"] Jul 08 02:17:16.371952 ip-10-0-134-84 kubenswrapper[2578]: W0708 02:17:16.371920 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76ea6f3_4365_4939_9cb2_ceaa2f950809.slice/crio-2a5832562d4073d8980113509d2bc049361163cd97df7b3a85fb0383be829783 WatchSource:0}: Error finding container 2a5832562d4073d8980113509d2bc049361163cd97df7b3a85fb0383be829783: Status 404 returned error can't find the container with id 2a5832562d4073d8980113509d2bc049361163cd97df7b3a85fb0383be829783 Jul 08 02:17:16.374125 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.374096 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:17:16.374205 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.374157 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:17:16.374205 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.374184 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:17:16.969424 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.969385 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" event={"ID":"d76ea6f3-4365-4939-9cb2-ceaa2f950809","Type":"ContainerStarted","Data":"77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee"} Jul 08 02:17:16.969853 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.969490 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" event={"ID":"d76ea6f3-4365-4939-9cb2-ceaa2f950809","Type":"ContainerStarted","Data":"2a5832562d4073d8980113509d2bc049361163cd97df7b3a85fb0383be829783"} Jul 08 02:17:16.989592 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:16.989546 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" podStartSLOduration=1.989532723 podStartE2EDuration="1.989532723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:17:16.988305439 +0000 UTC m=+2067.650219999" watchObservedRunningTime="2026-07-08 02:17:16.989532723 +0000 UTC m=+2067.651447260" Jul 08 02:17:17.237332 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:17.237246 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:17.242037 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:17.242014 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:17.973876 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:17.973791 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:17.974653 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:17.974630 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:18.811460 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:18.811425 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:17:29.914329 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:29.914298 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:17:32.975492 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:32.975455 2578 patch_prober.go:28] interesting pod/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 08 02:17:32.975895 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:32.975541 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 08 02:17:33.047292 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:33.047265 2578 generic.go:356] "Generic (PLEG): container finished" podID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerID="77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee" exitCode=0 Jul 08 02:17:33.047405 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:33.047339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" event={"ID":"d76ea6f3-4365-4939-9cb2-ceaa2f950809","Type":"ContainerDied","Data":"77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee"} Jul 08 02:17:33.047736 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:33.047720 2578 scope.go:117] "RemoveContainer" containerID="77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee" Jul 08 02:17:33.048692 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:33.048667 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:17:33.048789 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:33.048745 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:17:33.048852 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:33.048793 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:17:34.055486 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:34.055451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" event={"ID":"d76ea6f3-4365-4939-9cb2-ceaa2f950809","Type":"ContainerStarted","Data":"4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc"} Jul 08 02:17:34.237488 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:34.237432 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:34.237696 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:34.237582 2578 patch_prober.go:28] interesting pod/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 08 02:17:34.237696 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:34.237625 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 08 02:17:35.237212 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:35.236773 2578 patch_prober.go:28] interesting pod/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 08 02:17:35.237212 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:35.236832 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 08 02:17:36.237337 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:36.237295 2578 patch_prober.go:28] interesting pod/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 08 02:17:36.238404 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:36.237971 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 08 02:17:37.237635 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:37.237600 2578 patch_prober.go:28] interesting pod/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 08 02:17:37.238216 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:37.238187 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 08 02:17:38.240456 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:38.240423 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:38.240914 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:38.240706 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:38.241294 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:38.241279 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:17:39.717552 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.717516 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg"] Jul 08 02:17:39.720223 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.720200 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.722716 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.722689 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7c26aa\"/\"openshift-service-ca.crt\"" Jul 08 02:17:39.723761 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.723742 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7c26aa\"/\"default-dockercfg-q4nb2\"" Jul 08 02:17:39.723874 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.723782 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7c26aa\"/\"kube-root-ca.crt\"" Jul 08 02:17:39.723949 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.723869 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7c26aa\"/\"test-model-7c26aa-kserve-self-signed-certs\"" Jul 08 02:17:39.733018 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.732998 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg"] Jul 08 02:17:39.857852 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.857823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tmp-dir\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.858007 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.857876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-home\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.858007 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.857945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsnk\" (UniqueName: \"kubernetes.io/projected/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-kube-api-access-4vsnk\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.858007 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.857997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-model-cache\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.858114 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.858026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tls-certs\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.858152 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.858112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-dshm\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.959064 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.959033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-home\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.959243 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.959076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsnk\" (UniqueName: \"kubernetes.io/projected/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-kube-api-access-4vsnk\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.959243 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.959117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-model-cache\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.959243 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.959159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tls-certs\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.959368 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.959301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-dshm\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.959368 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.959353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tmp-dir\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.959557 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.959529 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-home\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.959711 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.959571 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-model-cache\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.959772 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.959716 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tmp-dir\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.961428 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.961410 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-dshm\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.961736 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.961717 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tls-certs\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:39.967287 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:39.967267 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsnk\" (UniqueName: \"kubernetes.io/projected/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-kube-api-access-4vsnk\") pod \"test-model-7c26aa-kserve-5b59576bb9-697qg\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:40.030937 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:40.030850 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:40.164553 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:40.164522 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg"] Jul 08 02:17:41.093758 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:41.093725 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" event={"ID":"f4cb6261-d1e1-446b-b9b3-27bba10e94a7","Type":"ContainerStarted","Data":"d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928"} Jul 08 02:17:41.094133 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:41.093766 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" event={"ID":"f4cb6261-d1e1-446b-b9b3-27bba10e94a7","Type":"ContainerStarted","Data":"b2d33adeb1985ff733d4749d6ffa86c78c291d6e6ec94ab353a596d9fea7fe0f"} Jul 08 02:17:41.110815 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:41.110773 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" podStartSLOduration=2.110758067 podStartE2EDuration="2.110758067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:17:41.109257061 +0000 UTC m=+2091.771171610" watchObservedRunningTime="2026-07-08 02:17:41.110758067 +0000 UTC m=+2091.772672646" Jul 08 02:17:43.207080 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:43.207047 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:17:50.031353 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:50.031325 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:50.031768 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:50.031372 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:50.043818 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:50.043793 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:50.145250 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:50.145207 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:17:54.511619 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:17:54.511588 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:18:13.942242 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:13.942210 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85"] Jul 08 02:18:13.945620 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:13.945602 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:13.948861 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:13.948830 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-219487\"/\"openshift-service-ca.crt\"" Jul 08 02:18:13.948861 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:13.948831 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-219487\"/\"default-dockercfg-h9d7v\"" Jul 08 02:18:13.949022 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:13.948897 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-219487\"/\"kube-root-ca.crt\"" Jul 08 02:18:13.949022 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:13.948832 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-219487\"/\"test-model-219487-kserve-self-signed-certs\"" Jul 08 02:18:13.956735 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:13.956715 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85"] Jul 08 02:18:14.064634 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.064606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2rg\" (UniqueName: \"kubernetes.io/projected/83d19711-d895-4c55-b44f-0fa951ea59bc-kube-api-access-9p2rg\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.064794 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.064652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-dshm\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.064794 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.064673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-tmp-dir\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.064794 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.064773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-home\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.064907 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.064793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-model-cache\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.064907 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.064819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d19711-d895-4c55-b44f-0fa951ea59bc-tls-certs\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.166322 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.166289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2rg\" (UniqueName: \"kubernetes.io/projected/83d19711-d895-4c55-b44f-0fa951ea59bc-kube-api-access-9p2rg\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.166322 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.166327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-dshm\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.166587 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.166345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-tmp-dir\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.166587 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.166386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-home\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.166587 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.166401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-model-cache\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.166587 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.166425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d19711-d895-4c55-b44f-0fa951ea59bc-tls-certs\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.166864 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.166838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-home\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.166930 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.166850 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-tmp-dir\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.166930 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.166889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-model-cache\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.168630 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.168609 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-dshm\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.168878 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.168861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d19711-d895-4c55-b44f-0fa951ea59bc-tls-certs\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.173796 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.173774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2rg\" (UniqueName: \"kubernetes.io/projected/83d19711-d895-4c55-b44f-0fa951ea59bc-kube-api-access-9p2rg\") pod \"test-model-219487-kserve-699585f595-d2k85\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.255124 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.255044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:14.386545 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:14.386520 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85"] Jul 08 02:18:14.389236 ip-10-0-134-84 kubenswrapper[2578]: W0708 02:18:14.389213 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83d19711_d895_4c55_b44f_0fa951ea59bc.slice/crio-2d9ce88967737223d43ea35da03d8f870beddc42f840e65a021a545e885b43ab WatchSource:0}: Error finding container 2d9ce88967737223d43ea35da03d8f870beddc42f840e65a021a545e885b43ab: Status 404 returned error can't find the container with id 2d9ce88967737223d43ea35da03d8f870beddc42f840e65a021a545e885b43ab Jul 08 02:18:15.249448 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:15.249408 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" event={"ID":"83d19711-d895-4c55-b44f-0fa951ea59bc","Type":"ContainerStarted","Data":"c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78"} Jul 08 02:18:15.249448 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:15.249450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" event={"ID":"83d19711-d895-4c55-b44f-0fa951ea59bc","Type":"ContainerStarted","Data":"2d9ce88967737223d43ea35da03d8f870beddc42f840e65a021a545e885b43ab"} Jul 08 02:18:15.267415 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:15.267367 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" podStartSLOduration=2.267350267 podStartE2EDuration="2.267350267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:18:15.265424624 +0000 UTC m=+2125.927339161" watchObservedRunningTime="2026-07-08 02:18:15.267350267 +0000 UTC m=+2125.929264803" Jul 08 02:18:19.226509 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:19.225505 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:18:24.255712 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:24.255671 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:24.256262 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:24.255726 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:24.268513 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:24.268466 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:18:24.302189 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:18:24.302159 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:19:00.629807 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.629728 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd"] Jul 08 02:19:00.632823 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.632806 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.636363 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.636267 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-f124b1\"/\"default-dockercfg-dpmqc\"" Jul 08 02:19:00.636363 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.636303 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-f124b1\"/\"openshift-service-ca.crt\"" Jul 08 02:19:00.636363 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.636302 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-f124b1\"/\"test-model-f124b1-kserve-self-signed-certs\"" Jul 08 02:19:00.636596 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.636380 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-f124b1\"/\"kube-root-ca.crt\"" Jul 08 02:19:00.641953 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.641932 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd"] Jul 08 02:19:00.772865 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.772831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89476d5e-821b-44f2-beac-f93a8c4570c4-tls-certs\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.773017 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.772894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-tmp-dir\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.773017 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.772953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-model-cache\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.773017 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.772991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-home\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.773122 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.773022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-dshm\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.773164 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.773142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kld2q\" (UniqueName: \"kubernetes.io/projected/89476d5e-821b-44f2-beac-f93a8c4570c4-kube-api-access-kld2q\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.874251 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.874216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89476d5e-821b-44f2-beac-f93a8c4570c4-tls-certs\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.874410 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.874374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-tmp-dir\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.874453 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.874425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-model-cache\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.874523 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.874468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-home\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.874568 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.874540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-dshm\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.874603 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.874592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kld2q\" (UniqueName: \"kubernetes.io/projected/89476d5e-821b-44f2-beac-f93a8c4570c4-kube-api-access-kld2q\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.874794 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.874772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-tmp-dir\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.874891 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.874834 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-model-cache\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.874956 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.874935 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-home\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.876636 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.876613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-dshm\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.876748 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.876732 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89476d5e-821b-44f2-beac-f93a8c4570c4-tls-certs\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.883638 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.883572 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kld2q\" (UniqueName: \"kubernetes.io/projected/89476d5e-821b-44f2-beac-f93a8c4570c4-kube-api-access-kld2q\") pod \"test-model-f124b1-kserve-6ffd4cdf85-67hwd\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:00.942660 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:00.942637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:01.074363 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:01.074336 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd"] Jul 08 02:19:01.076658 ip-10-0-134-84 kubenswrapper[2578]: W0708 02:19:01.076620 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89476d5e_821b_44f2_beac_f93a8c4570c4.slice/crio-3dc8d50dd8f2cc23c46695f3473eefa7faade74ae4ca005257fb14d6b868bc7e WatchSource:0}: Error finding container 3dc8d50dd8f2cc23c46695f3473eefa7faade74ae4ca005257fb14d6b868bc7e: Status 404 returned error can't find the container with id 3dc8d50dd8f2cc23c46695f3473eefa7faade74ae4ca005257fb14d6b868bc7e Jul 08 02:19:01.460421 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:01.460386 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" event={"ID":"89476d5e-821b-44f2-beac-f93a8c4570c4","Type":"ContainerStarted","Data":"b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553"} Jul 08 02:19:01.460604 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:01.460432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" event={"ID":"89476d5e-821b-44f2-beac-f93a8c4570c4","Type":"ContainerStarted","Data":"3dc8d50dd8f2cc23c46695f3473eefa7faade74ae4ca005257fb14d6b868bc7e"} Jul 08 02:19:01.476467 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:01.476422 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" podStartSLOduration=1.476408308 podStartE2EDuration="1.476408308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:19:01.476054131 +0000 UTC m=+2172.137968669" watchObservedRunningTime="2026-07-08 02:19:01.476408308 +0000 UTC m=+2172.138322845" Jul 08 02:19:04.612405 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:04.612373 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:19:10.943416 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:10.943378 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:10.943416 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:10.943425 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:10.955901 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:10.955878 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:11.519067 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:11.519038 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:22.125744 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:22.125709 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd"] Jul 08 02:19:27.613879 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:27.613847 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:19:37.147025 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.146981 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" podUID="89476d5e-821b-44f2-beac-f93a8c4570c4" containerName="main" containerID="cri-o://b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553" gracePeriod=45 Jul 08 02:19:37.393727 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.393699 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:37.512766 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.512685 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-model-cache\") pod \"89476d5e-821b-44f2-beac-f93a8c4570c4\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " Jul 08 02:19:37.512766 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.512748 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-dshm\") pod \"89476d5e-821b-44f2-beac-f93a8c4570c4\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " Jul 08 02:19:37.512959 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.512784 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89476d5e-821b-44f2-beac-f93a8c4570c4-tls-certs\") pod \"89476d5e-821b-44f2-beac-f93a8c4570c4\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " Jul 08 02:19:37.512959 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.512815 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-home\") pod \"89476d5e-821b-44f2-beac-f93a8c4570c4\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " Jul 08 02:19:37.512959 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.512833 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-tmp-dir\") pod \"89476d5e-821b-44f2-beac-f93a8c4570c4\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " Jul 08 02:19:37.513109 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.513000 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "89476d5e-821b-44f2-beac-f93a8c4570c4" (UID: "89476d5e-821b-44f2-beac-f93a8c4570c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:19:37.513109 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.513020 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kld2q\" (UniqueName: \"kubernetes.io/projected/89476d5e-821b-44f2-beac-f93a8c4570c4-kube-api-access-kld2q\") pod \"89476d5e-821b-44f2-beac-f93a8c4570c4\" (UID: \"89476d5e-821b-44f2-beac-f93a8c4570c4\") " Jul 08 02:19:37.513109 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.513091 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89476d5e-821b-44f2-beac-f93a8c4570c4" (UID: "89476d5e-821b-44f2-beac-f93a8c4570c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:19:37.513257 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.513111 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-home" (OuterVolumeSpecName: "home") pod "89476d5e-821b-44f2-beac-f93a8c4570c4" (UID: "89476d5e-821b-44f2-beac-f93a8c4570c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:19:37.513365 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.513348 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:19:37.513435 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.513366 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:19:37.513435 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.513375 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:19:37.515025 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.515000 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-dshm" (OuterVolumeSpecName: "dshm") pod "89476d5e-821b-44f2-beac-f93a8c4570c4" (UID: "89476d5e-821b-44f2-beac-f93a8c4570c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:19:37.515141 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.515035 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89476d5e-821b-44f2-beac-f93a8c4570c4-kube-api-access-kld2q" (OuterVolumeSpecName: "kube-api-access-kld2q") pod "89476d5e-821b-44f2-beac-f93a8c4570c4" (UID: "89476d5e-821b-44f2-beac-f93a8c4570c4"). InnerVolumeSpecName "kube-api-access-kld2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:19:37.515141 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.515095 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89476d5e-821b-44f2-beac-f93a8c4570c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89476d5e-821b-44f2-beac-f93a8c4570c4" (UID: "89476d5e-821b-44f2-beac-f93a8c4570c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:19:37.614231 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.614194 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89476d5e-821b-44f2-beac-f93a8c4570c4-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:19:37.614231 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.614230 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kld2q\" (UniqueName: \"kubernetes.io/projected/89476d5e-821b-44f2-beac-f93a8c4570c4-kube-api-access-kld2q\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:19:37.614231 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.614239 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89476d5e-821b-44f2-beac-f93a8c4570c4-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:19:37.630167 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.630140 2578 generic.go:356] "Generic (PLEG): container finished" podID="89476d5e-821b-44f2-beac-f93a8c4570c4" containerID="b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553" exitCode=0 Jul 08 02:19:37.630264 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.630171 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" event={"ID":"89476d5e-821b-44f2-beac-f93a8c4570c4","Type":"ContainerDied","Data":"b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553"} Jul 08 02:19:37.630264 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.630185 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" Jul 08 02:19:37.630264 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.630219 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd" event={"ID":"89476d5e-821b-44f2-beac-f93a8c4570c4","Type":"ContainerDied","Data":"3dc8d50dd8f2cc23c46695f3473eefa7faade74ae4ca005257fb14d6b868bc7e"} Jul 08 02:19:37.630264 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.630240 2578 scope.go:117] "RemoveContainer" containerID="b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553" Jul 08 02:19:37.639390 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.639375 2578 scope.go:117] "RemoveContainer" containerID="b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553" Jul 08 02:19:37.639687 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:19:37.639660 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553\": container with ID starting with b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553 not found: ID does not exist" containerID="b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553" Jul 08 02:19:37.639757 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.639700 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553"} err="failed to get container status \"b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553\": rpc error: code = NotFound desc = could not find container \"b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553\": container with ID starting with b4c5259313af8f168b210148d8cc034741911e25104b3315c9e4ad343c388553 not found: ID does not exist" Jul 08 02:19:37.652322 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.652301 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd"] Jul 08 02:19:37.656520 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.656501 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-reuse-f124b1/test-model-f124b1-kserve-6ffd4cdf85-67hwd"] Jul 08 02:19:37.906712 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:19:37.906687 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89476d5e-821b-44f2-beac-f93a8c4570c4" path="/var/lib/kubelet/pods/89476d5e-821b-44f2-beac-f93a8c4570c4/volumes" Jul 08 02:20:02.815406 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:02.815365 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:20:30.098164 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:30.098133 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg"] Jul 08 02:20:31.021986 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:31.021952 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85"] Jul 08 02:20:34.809367 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:34.809332 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:20:45.117812 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.117751 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" podUID="f4cb6261-d1e1-446b-b9b3-27bba10e94a7" containerName="main" containerID="cri-o://d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928" gracePeriod=45 Jul 08 02:20:45.367166 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.367138 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:20:45.413397 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413326 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-dshm\") pod \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " Jul 08 02:20:45.413571 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413404 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-model-cache\") pod \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " Jul 08 02:20:45.413571 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413430 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tmp-dir\") pod \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " Jul 08 02:20:45.413571 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413445 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tls-certs\") pod \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " Jul 08 02:20:45.413571 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413464 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vsnk\" (UniqueName: \"kubernetes.io/projected/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-kube-api-access-4vsnk\") pod \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " Jul 08 02:20:45.413571 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413540 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-home\") pod \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\" (UID: \"f4cb6261-d1e1-446b-b9b3-27bba10e94a7\") " Jul 08 02:20:45.413850 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413682 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4cb6261-d1e1-446b-b9b3-27bba10e94a7" (UID: "f4cb6261-d1e1-446b-b9b3-27bba10e94a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:20:45.413850 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413759 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4cb6261-d1e1-446b-b9b3-27bba10e94a7" (UID: "f4cb6261-d1e1-446b-b9b3-27bba10e94a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:20:45.413850 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413845 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:45.414006 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.413864 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-home" (OuterVolumeSpecName: "home") pod "f4cb6261-d1e1-446b-b9b3-27bba10e94a7" (UID: "f4cb6261-d1e1-446b-b9b3-27bba10e94a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:20:45.415658 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.415631 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-dshm" (OuterVolumeSpecName: "dshm") pod "f4cb6261-d1e1-446b-b9b3-27bba10e94a7" (UID: "f4cb6261-d1e1-446b-b9b3-27bba10e94a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:20:45.415776 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.415691 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4cb6261-d1e1-446b-b9b3-27bba10e94a7" (UID: "f4cb6261-d1e1-446b-b9b3-27bba10e94a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:20:45.415776 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.415708 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-kube-api-access-4vsnk" (OuterVolumeSpecName: "kube-api-access-4vsnk") pod "f4cb6261-d1e1-446b-b9b3-27bba10e94a7" (UID: "f4cb6261-d1e1-446b-b9b3-27bba10e94a7"). InnerVolumeSpecName "kube-api-access-4vsnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:20:45.514502 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.514433 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:45.514502 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.514497 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:45.514502 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.514513 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsnk\" (UniqueName: \"kubernetes.io/projected/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-kube-api-access-4vsnk\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:45.514722 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.514526 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:45.514722 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.514537 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cb6261-d1e1-446b-b9b3-27bba10e94a7-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:45.937909 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.937876 2578 generic.go:356] "Generic (PLEG): container finished" podID="f4cb6261-d1e1-446b-b9b3-27bba10e94a7" containerID="d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928" exitCode=0 Jul 08 02:20:45.938066 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.937915 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" Jul 08 02:20:45.938066 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.937960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" event={"ID":"f4cb6261-d1e1-446b-b9b3-27bba10e94a7","Type":"ContainerDied","Data":"d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928"} Jul 08 02:20:45.938066 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.938002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg" event={"ID":"f4cb6261-d1e1-446b-b9b3-27bba10e94a7","Type":"ContainerDied","Data":"b2d33adeb1985ff733d4749d6ffa86c78c291d6e6ec94ab353a596d9fea7fe0f"} Jul 08 02:20:45.938066 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.938025 2578 scope.go:117] "RemoveContainer" containerID="d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928" Jul 08 02:20:45.946328 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.946313 2578 scope.go:117] "RemoveContainer" containerID="d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928" Jul 08 02:20:45.946589 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:20:45.946567 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928\": container with ID starting with d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928 not found: ID does not exist" containerID="d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928" Jul 08 02:20:45.946631 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.946597 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928"} err="failed to get container status \"d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928\": rpc error: code = NotFound desc = could not find container \"d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928\": container with ID starting with d2474932bad1c205f1b53834dfda2157ff3148efb11078375a7dc4bd32cdb928 not found: ID does not exist" Jul 08 02:20:45.954934 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.954910 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg"] Jul 08 02:20:45.958411 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:45.958391 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7c26aa/test-model-7c26aa-kserve-5b59576bb9-697qg"] Jul 08 02:20:46.041919 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.041873 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" podUID="83d19711-d895-4c55-b44f-0fa951ea59bc" containerName="main" containerID="cri-o://c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78" gracePeriod=45 Jul 08 02:20:46.286470 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.286447 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:20:46.321266 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321240 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-home\") pod \"83d19711-d895-4c55-b44f-0fa951ea59bc\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " Jul 08 02:20:46.321416 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321327 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-tmp-dir\") pod \"83d19711-d895-4c55-b44f-0fa951ea59bc\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " Jul 08 02:20:46.321416 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321360 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d19711-d895-4c55-b44f-0fa951ea59bc-tls-certs\") pod \"83d19711-d895-4c55-b44f-0fa951ea59bc\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " Jul 08 02:20:46.321416 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321399 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p2rg\" (UniqueName: \"kubernetes.io/projected/83d19711-d895-4c55-b44f-0fa951ea59bc-kube-api-access-9p2rg\") pod \"83d19711-d895-4c55-b44f-0fa951ea59bc\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " Jul 08 02:20:46.321606 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321422 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-dshm\") pod \"83d19711-d895-4c55-b44f-0fa951ea59bc\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " Jul 08 02:20:46.321606 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321472 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-home" (OuterVolumeSpecName: "home") pod "83d19711-d895-4c55-b44f-0fa951ea59bc" (UID: "83d19711-d895-4c55-b44f-0fa951ea59bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:20:46.321606 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321496 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-model-cache\") pod \"83d19711-d895-4c55-b44f-0fa951ea59bc\" (UID: \"83d19711-d895-4c55-b44f-0fa951ea59bc\") " Jul 08 02:20:46.321765 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321607 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83d19711-d895-4c55-b44f-0fa951ea59bc" (UID: "83d19711-d895-4c55-b44f-0fa951ea59bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:20:46.321822 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321767 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "83d19711-d895-4c55-b44f-0fa951ea59bc" (UID: "83d19711-d895-4c55-b44f-0fa951ea59bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:20:46.321862 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321850 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:46.321925 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321870 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:46.321925 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.321884 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:46.323709 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.323681 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-dshm" (OuterVolumeSpecName: "dshm") pod "83d19711-d895-4c55-b44f-0fa951ea59bc" (UID: "83d19711-d895-4c55-b44f-0fa951ea59bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:20:46.323805 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.323717 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d19711-d895-4c55-b44f-0fa951ea59bc-kube-api-access-9p2rg" (OuterVolumeSpecName: "kube-api-access-9p2rg") pod "83d19711-d895-4c55-b44f-0fa951ea59bc" (UID: "83d19711-d895-4c55-b44f-0fa951ea59bc"). InnerVolumeSpecName "kube-api-access-9p2rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:20:46.323805 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.323752 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d19711-d895-4c55-b44f-0fa951ea59bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83d19711-d895-4c55-b44f-0fa951ea59bc" (UID: "83d19711-d895-4c55-b44f-0fa951ea59bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 02:20:46.423111 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.423080 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d19711-d895-4c55-b44f-0fa951ea59bc-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:46.423111 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.423107 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p2rg\" (UniqueName: \"kubernetes.io/projected/83d19711-d895-4c55-b44f-0fa951ea59bc-kube-api-access-9p2rg\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:46.423279 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.423117 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d19711-d895-4c55-b44f-0fa951ea59bc-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:20:46.944703 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.944671 2578 generic.go:356] "Generic (PLEG): container finished" podID="83d19711-d895-4c55-b44f-0fa951ea59bc" containerID="c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78" exitCode=0 Jul 08 02:20:46.944864 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.944714 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" Jul 08 02:20:46.944864 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.944762 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" event={"ID":"83d19711-d895-4c55-b44f-0fa951ea59bc","Type":"ContainerDied","Data":"c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78"} Jul 08 02:20:46.944864 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.944807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85" event={"ID":"83d19711-d895-4c55-b44f-0fa951ea59bc","Type":"ContainerDied","Data":"2d9ce88967737223d43ea35da03d8f870beddc42f840e65a021a545e885b43ab"} Jul 08 02:20:46.944864 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.944826 2578 scope.go:117] "RemoveContainer" containerID="c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78" Jul 08 02:20:46.953883 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.953865 2578 scope.go:117] "RemoveContainer" containerID="c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78" Jul 08 02:20:46.954125 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:20:46.954106 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78\": container with ID starting with c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78 not found: ID does not exist" containerID="c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78" Jul 08 02:20:46.954177 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.954134 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78"} err="failed to get container status \"c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78\": rpc error: code = NotFound desc = could not find container \"c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78\": container with ID starting with c61b87faceda1fbbe362100782c983cd444a337600b671152458225b23456f78 not found: ID does not exist" Jul 08 02:20:46.966148 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.966123 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85"] Jul 08 02:20:46.970670 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:46.970646 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-219487/test-model-219487-kserve-699585f595-d2k85"] Jul 08 02:20:47.906308 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:47.906276 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d19711-d895-4c55-b44f-0fa951ea59bc" path="/var/lib/kubelet/pods/83d19711-d895-4c55-b44f-0fa951ea59bc/volumes" Jul 08 02:20:47.906821 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:20:47.906805 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4cb6261-d1e1-446b-b9b3-27bba10e94a7" path="/var/lib/kubelet/pods/f4cb6261-d1e1-446b-b9b3-27bba10e94a7/volumes" Jul 08 02:21:17.710824 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:17.710783 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:21:20.669615 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:20.669579 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl"] Jul 08 02:21:20.670055 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:20.669787 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" containerID="cri-o://4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc" gracePeriod=30 Jul 08 02:21:23.243100 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:23.243064 2578 patch_prober.go:28] interesting pod/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 02:21:23.243459 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:23.243165 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 02:21:25.922962 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.922937 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:21:25.985782 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.985706 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-envoy\") pod \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " Jul 08 02:21:25.985782 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.985742 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-token\") pod \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " Jul 08 02:21:25.985782 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.985772 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-socket\") pod \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " Jul 08 02:21:25.986041 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.985788 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istiod-ca-cert\") pod \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " Jul 08 02:21:25.986041 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.985822 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-data\") pod \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " Jul 08 02:21:25.986041 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.985837 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-certs\") pod \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " Jul 08 02:21:25.986041 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.985864 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltwbv\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-kube-api-access-ltwbv\") pod \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " Jul 08 02:21:25.986041 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.985898 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-credential-socket\") pod \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " Jul 08 02:21:25.986041 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.985926 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-podinfo\") pod \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\" (UID: \"d76ea6f3-4365-4939-9cb2-ceaa2f950809\") " Jul 08 02:21:25.986041 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.986025 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d76ea6f3-4365-4939-9cb2-ceaa2f950809" (UID: "d76ea6f3-4365-4939-9cb2-ceaa2f950809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:21:25.986391 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.986223 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-socket\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:21:25.986391 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.986225 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d76ea6f3-4365-4939-9cb2-ceaa2f950809" (UID: "d76ea6f3-4365-4939-9cb2-ceaa2f950809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:21:25.986391 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.986235 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d76ea6f3-4365-4939-9cb2-ceaa2f950809" (UID: "d76ea6f3-4365-4939-9cb2-ceaa2f950809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:21:25.986570 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.986522 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d76ea6f3-4365-4939-9cb2-ceaa2f950809" (UID: "d76ea6f3-4365-4939-9cb2-ceaa2f950809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:21:25.986678 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.986656 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-data" (OuterVolumeSpecName: "istio-data") pod "d76ea6f3-4365-4939-9cb2-ceaa2f950809" (UID: "d76ea6f3-4365-4939-9cb2-ceaa2f950809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:21:25.988048 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.988014 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d76ea6f3-4365-4939-9cb2-ceaa2f950809" (UID: "d76ea6f3-4365-4939-9cb2-ceaa2f950809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:21:25.988180 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.988138 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d76ea6f3-4365-4939-9cb2-ceaa2f950809" (UID: "d76ea6f3-4365-4939-9cb2-ceaa2f950809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 02:21:25.988315 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.988295 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-token" (OuterVolumeSpecName: "istio-token") pod "d76ea6f3-4365-4939-9cb2-ceaa2f950809" (UID: "d76ea6f3-4365-4939-9cb2-ceaa2f950809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:21:25.988361 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:25.988344 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-kube-api-access-ltwbv" (OuterVolumeSpecName: "kube-api-access-ltwbv") pod "d76ea6f3-4365-4939-9cb2-ceaa2f950809" (UID: "d76ea6f3-4365-4939-9cb2-ceaa2f950809"). InnerVolumeSpecName "kube-api-access-ltwbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:21:26.087157 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.087118 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-data\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:21:26.087157 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.087152 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-workload-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:21:26.087328 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.087180 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltwbv\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-kube-api-access-ltwbv\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:21:26.087328 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.087191 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-credential-socket\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:21:26.087328 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.087199 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-podinfo\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:21:26.087328 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.087214 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-envoy\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:21:26.087328 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.087222 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istio-token\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:21:26.087328 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.087230 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76ea6f3-4365-4939-9cb2-ceaa2f950809-istiod-ca-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:21:26.118362 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.118330 2578 generic.go:356] "Generic (PLEG): container finished" podID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerID="4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc" exitCode=0 Jul 08 02:21:26.118448 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.118377 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" Jul 08 02:21:26.118448 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.118420 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" event={"ID":"d76ea6f3-4365-4939-9cb2-ceaa2f950809","Type":"ContainerDied","Data":"4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc"} Jul 08 02:21:26.118542 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.118472 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl" event={"ID":"d76ea6f3-4365-4939-9cb2-ceaa2f950809","Type":"ContainerDied","Data":"2a5832562d4073d8980113509d2bc049361163cd97df7b3a85fb0383be829783"} Jul 08 02:21:26.118542 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.118510 2578 scope.go:117] "RemoveContainer" containerID="4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc" Jul 08 02:21:26.127407 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.127390 2578 scope.go:117] "RemoveContainer" containerID="77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee" Jul 08 02:21:26.135171 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.135152 2578 scope.go:117] "RemoveContainer" containerID="4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc" Jul 08 02:21:26.135410 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:21:26.135392 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc\": container with ID starting with 4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc not found: ID does not exist" containerID="4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc" Jul 08 02:21:26.135453 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.135418 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc"} err="failed to get container status \"4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc\": rpc error: code = NotFound desc = could not find container \"4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc\": container with ID starting with 4773c8c1b8f042384a8b9f545a0fd2f3596467a616127647ddf846fde23cf8dc not found: ID does not exist" Jul 08 02:21:26.135453 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.135434 2578 scope.go:117] "RemoveContainer" containerID="77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee" Jul 08 02:21:26.135678 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:21:26.135660 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee\": container with ID starting with 77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee not found: ID does not exist" containerID="77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee" Jul 08 02:21:26.135724 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.135692 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee"} err="failed to get container status \"77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee\": rpc error: code = NotFound desc = could not find container \"77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee\": container with ID starting with 77858406ef8eb9954da4a3c8ff277e27cb90760d75d196064a8d8c5ebd0129ee not found: ID does not exist" Jul 08 02:21:26.139747 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.139714 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl"] Jul 08 02:21:26.141427 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:26.141408 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-219487-openshift-default-5bbddd6cbf-4sfrl"] Jul 08 02:21:27.906548 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:27.906515 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" path="/var/lib/kubelet/pods/d76ea6f3-4365-4939-9cb2-ceaa2f950809/volumes" Jul 08 02:21:29.308454 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308421 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2"] Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308812 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4cb6261-d1e1-446b-b9b3-27bba10e94a7" containerName="main" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308825 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4cb6261-d1e1-446b-b9b3-27bba10e94a7" containerName="main" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308835 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308842 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308850 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308856 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308862 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d19711-d895-4c55-b44f-0fa951ea59bc" containerName="main" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308867 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d19711-d895-4c55-b44f-0fa951ea59bc" containerName="main" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308887 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89476d5e-821b-44f2-beac-f93a8c4570c4" containerName="main" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308895 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="89476d5e-821b-44f2-beac-f93a8c4570c4" containerName="main" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308979 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.308991 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="89476d5e-821b-44f2-beac-f93a8c4570c4" containerName="main" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.309001 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4cb6261-d1e1-446b-b9b3-27bba10e94a7" containerName="main" Jul 08 02:21:29.309162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.309016 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d19711-d895-4c55-b44f-0fa951ea59bc" containerName="main" Jul 08 02:21:29.312065 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.312048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.314742 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.314711 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1df4d7-openshift-default-dockercfg-rn8vl\"" Jul 08 02:21:29.326814 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.326787 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2"] Jul 08 02:21:29.415552 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.415512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbxtc\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-kube-api-access-kbxtc\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.415763 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.415584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-data\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.415763 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.415611 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-socket\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.415763 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.415646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-token\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.415763 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.415695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-certs\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.415763 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.415720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-credential-socket\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.416009 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.415769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-podinfo\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.416009 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.415798 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-envoy\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.416009 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.415873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istiod-ca-cert\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517339 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-podinfo\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517339 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-envoy\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517606 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istiod-ca-cert\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517606 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbxtc\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-kube-api-access-kbxtc\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517606 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-data\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517777 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-socket\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517777 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-token\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517777 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-certs\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517930 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-credential-socket\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.517930 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517910 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-data\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.518037 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.517950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-socket\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.518095 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.518075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-certs\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.518279 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.518258 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istiod-ca-cert\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.518410 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.518391 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-credential-socket\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.519618 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.519601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-envoy\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.519836 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.519817 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-podinfo\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.524955 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.524930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-token\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.525062 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.525045 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbxtc\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-kube-api-access-kbxtc\") pod \"e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.622255 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.622218 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:29.756953 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.756920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2"] Jul 08 02:21:29.759799 ip-10-0-134-84 kubenswrapper[2578]: W0708 02:21:29.759772 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528b0ce7_2144_409e_a9c7_0e595fd3b3c5.slice/crio-a5ee0e0cf98397141085203cc842415593b0d93b1afd46e75758eb47d6b092df WatchSource:0}: Error finding container a5ee0e0cf98397141085203cc842415593b0d93b1afd46e75758eb47d6b092df: Status 404 returned error can't find the container with id a5ee0e0cf98397141085203cc842415593b0d93b1afd46e75758eb47d6b092df Jul 08 02:21:29.764823 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.764787 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:21:29.764885 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.764873 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:21:29.764927 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:29.764911 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:21:30.137608 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:30.137573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" event={"ID":"528b0ce7-2144-409e-a9c7-0e595fd3b3c5","Type":"ContainerStarted","Data":"58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0"} Jul 08 02:21:30.137797 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:30.137630 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" event={"ID":"528b0ce7-2144-409e-a9c7-0e595fd3b3c5","Type":"ContainerStarted","Data":"a5ee0e0cf98397141085203cc842415593b0d93b1afd46e75758eb47d6b092df"} Jul 08 02:21:30.158020 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:30.157966 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" podStartSLOduration=1.157949827 podStartE2EDuration="1.157949827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:21:30.155987299 +0000 UTC m=+2320.817901858" watchObservedRunningTime="2026-07-08 02:21:30.157949827 +0000 UTC m=+2320.819864364" Jul 08 02:21:30.622859 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:30.622816 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:30.627912 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:30.627887 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:31.142830 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:31.142800 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:31.143932 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:31.143915 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:32.922663 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:32.922635 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:21:45.542319 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:45.542226 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:21:47.185618 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:47.185581 2578 patch_prober.go:28] interesting pod/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 02:21:47.185987 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:47.185645 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 02:21:49.227595 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:49.227565 2578 generic.go:356] "Generic (PLEG): container finished" podID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerID="58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0" exitCode=0 Jul 08 02:21:49.228051 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:49.227648 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" event={"ID":"528b0ce7-2144-409e-a9c7-0e595fd3b3c5","Type":"ContainerDied","Data":"58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0"} Jul 08 02:21:49.228098 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:49.228047 2578 scope.go:117] "RemoveContainer" containerID="58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0" Jul 08 02:21:49.228980 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:49.228955 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:21:49.229040 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:49.229020 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:21:49.229075 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:49.229055 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:21:50.237576 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:50.237538 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" event={"ID":"528b0ce7-2144-409e-a9c7-0e595fd3b3c5","Type":"ContainerStarted","Data":"764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85"} Jul 08 02:21:50.622565 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:50.622521 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:50.622776 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:50.622753 2578 patch_prober.go:28] interesting pod/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 08 02:21:50.622837 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:50.622817 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 08 02:21:51.622719 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:51.622681 2578 patch_prober.go:28] interesting pod/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 08 02:21:51.623146 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:51.622743 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 08 02:21:52.622816 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:52.622780 2578 patch_prober.go:28] interesting pod/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 08 02:21:52.623230 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:52.622837 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 08 02:21:53.623248 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:53.623216 2578 patch_prober.go:28] interesting pod/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 08 02:21:53.623658 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:53.623269 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 08 02:21:54.626773 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:54.626742 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:54.627173 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:54.626997 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:54.627627 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:54.627609 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:21:56.408360 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:56.408326 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:21:58.455386 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:58.455353 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2"] Jul 08 02:21:58.455833 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:21:58.455602 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" containerID="cri-o://764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85" gracePeriod=30 Jul 08 02:22:03.709060 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.709037 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:22:03.748236 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748169 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istiod-ca-cert\") pod \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " Jul 08 02:22:03.748236 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748211 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-socket\") pod \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " Jul 08 02:22:03.748443 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748242 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-credential-socket\") pod \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " Jul 08 02:22:03.748443 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748269 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-data\") pod \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " Jul 08 02:22:03.748443 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748321 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-envoy\") pod \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " Jul 08 02:22:03.748443 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748346 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-token\") pod \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " Jul 08 02:22:03.748443 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748369 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbxtc\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-kube-api-access-kbxtc\") pod \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " Jul 08 02:22:03.748443 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748440 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-certs\") pod \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " Jul 08 02:22:03.748801 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748513 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-podinfo\") pod \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\" (UID: \"528b0ce7-2144-409e-a9c7-0e595fd3b3c5\") " Jul 08 02:22:03.748801 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748557 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "528b0ce7-2144-409e-a9c7-0e595fd3b3c5" (UID: "528b0ce7-2144-409e-a9c7-0e595fd3b3c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:22:03.748801 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748611 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "528b0ce7-2144-409e-a9c7-0e595fd3b3c5" (UID: "528b0ce7-2144-409e-a9c7-0e595fd3b3c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:22:03.748801 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748649 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "528b0ce7-2144-409e-a9c7-0e595fd3b3c5" (UID: "528b0ce7-2144-409e-a9c7-0e595fd3b3c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:22:03.748999 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748827 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "528b0ce7-2144-409e-a9c7-0e595fd3b3c5" (UID: "528b0ce7-2144-409e-a9c7-0e595fd3b3c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:22:03.748999 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748865 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istiod-ca-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:22:03.748999 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748883 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-socket\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:22:03.748999 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.748896 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-credential-socket\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:22:03.749613 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.749581 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "528b0ce7-2144-409e-a9c7-0e595fd3b3c5" (UID: "528b0ce7-2144-409e-a9c7-0e595fd3b3c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:22:03.750746 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.750716 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "528b0ce7-2144-409e-a9c7-0e595fd3b3c5" (UID: "528b0ce7-2144-409e-a9c7-0e595fd3b3c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:22:03.750843 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.750778 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "528b0ce7-2144-409e-a9c7-0e595fd3b3c5" (UID: "528b0ce7-2144-409e-a9c7-0e595fd3b3c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:22:03.751076 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.751053 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-kube-api-access-kbxtc" (OuterVolumeSpecName: "kube-api-access-kbxtc") pod "528b0ce7-2144-409e-a9c7-0e595fd3b3c5" (UID: "528b0ce7-2144-409e-a9c7-0e595fd3b3c5"). InnerVolumeSpecName "kube-api-access-kbxtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:22:03.751076 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.751064 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "528b0ce7-2144-409e-a9c7-0e595fd3b3c5" (UID: "528b0ce7-2144-409e-a9c7-0e595fd3b3c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 02:22:03.850214 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.850187 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-workload-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:22:03.850214 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.850213 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-podinfo\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:22:03.850365 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.850226 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-data\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:22:03.850365 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.850237 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-envoy\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:22:03.850365 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.850249 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-istio-token\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:22:03.850365 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:03.850261 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbxtc\" (UniqueName: \"kubernetes.io/projected/528b0ce7-2144-409e-a9c7-0e595fd3b3c5-kube-api-access-kbxtc\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:22:04.306012 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.305980 2578 generic.go:356] "Generic (PLEG): container finished" podID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerID="764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85" exitCode=0 Jul 08 02:22:04.306174 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.306016 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" event={"ID":"528b0ce7-2144-409e-a9c7-0e595fd3b3c5","Type":"ContainerDied","Data":"764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85"} Jul 08 02:22:04.306174 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.306040 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" Jul 08 02:22:04.306174 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.306056 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2" event={"ID":"528b0ce7-2144-409e-a9c7-0e595fd3b3c5","Type":"ContainerDied","Data":"a5ee0e0cf98397141085203cc842415593b0d93b1afd46e75758eb47d6b092df"} Jul 08 02:22:04.306174 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.306077 2578 scope.go:117] "RemoveContainer" containerID="764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85" Jul 08 02:22:04.314640 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.314607 2578 scope.go:117] "RemoveContainer" containerID="58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0" Jul 08 02:22:04.322126 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.322109 2578 scope.go:117] "RemoveContainer" containerID="764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85" Jul 08 02:22:04.322355 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:22:04.322332 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85\": container with ID starting with 764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85 not found: ID does not exist" containerID="764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85" Jul 08 02:22:04.322431 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.322362 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85"} err="failed to get container status \"764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85\": rpc error: code = NotFound desc = could not find container \"764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85\": container with ID starting with 764e2cb7fe334a59fcd09da10e4f1c96f36ce947f3a4ad33fe862584f91c9d85 not found: ID does not exist" Jul 08 02:22:04.322431 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.322379 2578 scope.go:117] "RemoveContainer" containerID="58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0" Jul 08 02:22:04.322858 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:22:04.322832 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0\": container with ID starting with 58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0 not found: ID does not exist" containerID="58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0" Jul 08 02:22:04.322970 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.322865 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0"} err="failed to get container status \"58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0\": rpc error: code = NotFound desc = could not find container \"58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0\": container with ID starting with 58896c720dded2c2bfb9d6391db12420c336b91679609374f9f53c095cb898e0 not found: ID does not exist" Jul 08 02:22:04.325447 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.325423 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2"] Jul 08 02:22:04.327282 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:04.327257 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1df4d7-openshift-default-8575bf77cc-xf5r2"] Jul 08 02:22:05.906650 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:05.906620 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" path="/var/lib/kubelet/pods/528b0ce7-2144-409e-a9c7-0e595fd3b3c5/volumes" Jul 08 02:22:07.368742 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.368712 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m"] Jul 08 02:22:07.369098 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.369041 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" Jul 08 02:22:07.369098 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.369052 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" Jul 08 02:22:07.369098 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.369064 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" Jul 08 02:22:07.369098 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.369070 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" Jul 08 02:22:07.369248 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.369125 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d76ea6f3-4365-4939-9cb2-ceaa2f950809" containerName="istio-proxy" Jul 08 02:22:07.369248 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.369132 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" Jul 08 02:22:07.372086 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.372063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.374374 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.374354 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b8b505-openshift-default-dockercfg-pf76q\"" Jul 08 02:22:07.384187 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.384163 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m"] Jul 08 02:22:07.484184 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.484150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-token\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.484184 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.484186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-podinfo\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.484398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.484210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-certs\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.484398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.484289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-data\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.484398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.484310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmqh\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-kube-api-access-fxmqh\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.484398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.484333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istiod-ca-cert\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.484398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.484370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-credential-socket\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.484398 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.484387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-envoy\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.484611 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.484452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-socket\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.585764 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.585730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-socket\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.585940 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.585802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-token\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.585940 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.585822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-podinfo\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.585940 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.585844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-certs\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.585940 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.585876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-data\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.585940 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.585892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmqh\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-kube-api-access-fxmqh\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.585940 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.585915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istiod-ca-cert\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.586269 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.585946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-credential-socket\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.586269 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.585970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-envoy\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.586269 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.586184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-socket\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.586389 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.586269 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-certs\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.586389 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.586316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-data\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.586389 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.586342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-credential-socket\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.586706 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.586681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istiod-ca-cert\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.588198 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.588173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-envoy\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.588282 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.588261 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-podinfo\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.595558 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.595539 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-token\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.595858 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.595840 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmqh\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-kube-api-access-fxmqh\") pod \"e2e-shared-b-b8b505-openshift-default-798484f767-24w8m\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.681851 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.681769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:07.815803 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.815777 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m"] Jul 08 02:22:07.818289 ip-10-0-134-84 kubenswrapper[2578]: W0708 02:22:07.818257 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96643d42_8630_4bd0_b5e9_a4ee97dfedc9.slice/crio-718ae6399899b042acf0867a0ee153be19ccdf87e46e96d249e86f52bc9d6cbf WatchSource:0}: Error finding container 718ae6399899b042acf0867a0ee153be19ccdf87e46e96d249e86f52bc9d6cbf: Status 404 returned error can't find the container with id 718ae6399899b042acf0867a0ee153be19ccdf87e46e96d249e86f52bc9d6cbf Jul 08 02:22:07.820594 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.820565 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:22:07.820670 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.820628 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:22:07.820670 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:07.820656 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:22:08.211527 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:08.211488 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:22:08.333328 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:08.333294 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" event={"ID":"96643d42-8630-4bd0-b5e9-a4ee97dfedc9","Type":"ContainerStarted","Data":"8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac"} Jul 08 02:22:08.333520 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:08.333371 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" event={"ID":"96643d42-8630-4bd0-b5e9-a4ee97dfedc9","Type":"ContainerStarted","Data":"718ae6399899b042acf0867a0ee153be19ccdf87e46e96d249e86f52bc9d6cbf"} Jul 08 02:22:08.354837 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:08.354785 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" podStartSLOduration=1.354769051 podStartE2EDuration="1.354769051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:22:08.353217646 +0000 UTC m=+2359.015132183" watchObservedRunningTime="2026-07-08 02:22:08.354769051 +0000 UTC m=+2359.016683589" Jul 08 02:22:08.681989 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:08.681952 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:08.686849 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:08.686823 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:09.338391 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:09.338357 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:09.339328 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:09.339309 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:20.919893 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:20.919859 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:22:22.397852 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:22.397813 2578 generic.go:356] "Generic (PLEG): container finished" podID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerID="8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac" exitCode=0 Jul 08 02:22:22.398304 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:22.397885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" event={"ID":"96643d42-8630-4bd0-b5e9-a4ee97dfedc9","Type":"ContainerDied","Data":"8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac"} Jul 08 02:22:22.398304 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:22.398280 2578 scope.go:117] "RemoveContainer" containerID="8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac" Jul 08 02:22:22.399303 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:22.399275 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:22:22.399407 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:22.399341 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:22:22.399407 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:22.399369 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 02:22:23.403796 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:23.403760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" event={"ID":"96643d42-8630-4bd0-b5e9-a4ee97dfedc9","Type":"ContainerStarted","Data":"71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b"} Jul 08 02:22:23.682974 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:23.682878 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:23.684198 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:23.684175 2578 patch_prober.go:28] interesting pod/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 08 02:22:23.684262 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:23.684224 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 08 02:22:24.338675 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:24.338633 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:24.682360 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:24.682328 2578 patch_prober.go:28] interesting pod/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 08 02:22:24.682776 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:24.682381 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 08 02:22:25.682400 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:25.682366 2578 patch_prober.go:28] interesting pod/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 08 02:22:25.682823 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:25.682420 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 08 02:22:26.682771 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:26.682734 2578 patch_prober.go:28] interesting pod/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 08 02:22:26.683179 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:26.682788 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 08 02:22:27.686960 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:27.686926 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:22:27.688050 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:22:27.688027 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:25:23.701205 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:23.701172 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:25:33.189321 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:33.189284 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m"] Jul 08 02:25:33.189734 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:33.189569 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" containerID="cri-o://71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b" gracePeriod=30 Jul 08 02:25:33.946177 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:33.946144 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575bbf8489-jq5t7_e311fc41-f7c2-4f54-8dee-c577ee2a6942/authorino/0.log" Jul 08 02:25:35.207532 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:35.207494 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jvbzf"] Jul 08 02:25:38.445167 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.445143 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:25:38.535993 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.535921 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-m2bv4_bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf/manager/0.log" Jul 08 02:25:38.545194 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545170 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istiod-ca-cert\") pod \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " Jul 08 02:25:38.545300 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545202 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-data\") pod \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " Jul 08 02:25:38.545300 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545238 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-credential-socket\") pod \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " Jul 08 02:25:38.545404 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545368 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-token\") pod \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " Jul 08 02:25:38.545458 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545416 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-socket\") pod \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " Jul 08 02:25:38.545540 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545501 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-certs\") pod \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " Jul 08 02:25:38.545540 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545513 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96643d42-8630-4bd0-b5e9-a4ee97dfedc9" (UID: "96643d42-8630-4bd0-b5e9-a4ee97dfedc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:25:38.545540 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545533 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-envoy\") pod \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " Jul 08 02:25:38.545693 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545587 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-podinfo\") pod \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " Jul 08 02:25:38.545693 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545592 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96643d42-8630-4bd0-b5e9-a4ee97dfedc9" (UID: "96643d42-8630-4bd0-b5e9-a4ee97dfedc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 02:25:38.545693 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545612 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxmqh\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-kube-api-access-fxmqh\") pod \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\" (UID: \"96643d42-8630-4bd0-b5e9-a4ee97dfedc9\") " Jul 08 02:25:38.545843 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545728 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96643d42-8630-4bd0-b5e9-a4ee97dfedc9" (UID: "96643d42-8630-4bd0-b5e9-a4ee97dfedc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:25:38.546015 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.545991 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istiod-ca-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:25:38.546015 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.546017 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-credential-socket\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:25:38.546187 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.546033 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-socket\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:25:38.546187 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.546051 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96643d42-8630-4bd0-b5e9-a4ee97dfedc9" (UID: "96643d42-8630-4bd0-b5e9-a4ee97dfedc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:25:38.546291 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.546249 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "96643d42-8630-4bd0-b5e9-a4ee97dfedc9" (UID: "96643d42-8630-4bd0-b5e9-a4ee97dfedc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:25:38.548101 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.548075 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "96643d42-8630-4bd0-b5e9-a4ee97dfedc9" (UID: "96643d42-8630-4bd0-b5e9-a4ee97dfedc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:25:38.548192 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.548094 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-kube-api-access-fxmqh" (OuterVolumeSpecName: "kube-api-access-fxmqh") pod "96643d42-8630-4bd0-b5e9-a4ee97dfedc9" (UID: "96643d42-8630-4bd0-b5e9-a4ee97dfedc9"). InnerVolumeSpecName "kube-api-access-fxmqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 02:25:38.548192 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.548110 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96643d42-8630-4bd0-b5e9-a4ee97dfedc9" (UID: "96643d42-8630-4bd0-b5e9-a4ee97dfedc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 02:25:38.548192 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.548126 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96643d42-8630-4bd0-b5e9-a4ee97dfedc9" (UID: "96643d42-8630-4bd0-b5e9-a4ee97dfedc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 02:25:38.646909 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.646876 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-data\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:25:38.646909 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.646902 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-token\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:25:38.646909 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.646915 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-workload-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:25:38.647125 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.646927 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-envoy\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:25:38.647125 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.646938 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-istio-podinfo\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:25:38.647125 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:38.646949 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxmqh\" (UniqueName: \"kubernetes.io/projected/96643d42-8630-4bd0-b5e9-a4ee97dfedc9-kube-api-access-fxmqh\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\"" Jul 08 02:25:39.287015 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.286983 2578 generic.go:356] "Generic (PLEG): container finished" podID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerID="71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b" exitCode=0 Jul 08 02:25:39.287211 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.287031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" event={"ID":"96643d42-8630-4bd0-b5e9-a4ee97dfedc9","Type":"ContainerDied","Data":"71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b"} Jul 08 02:25:39.287211 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.287039 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" Jul 08 02:25:39.287211 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.287087 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m" event={"ID":"96643d42-8630-4bd0-b5e9-a4ee97dfedc9","Type":"ContainerDied","Data":"718ae6399899b042acf0867a0ee153be19ccdf87e46e96d249e86f52bc9d6cbf"} Jul 08 02:25:39.287211 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.287116 2578 scope.go:117] "RemoveContainer" containerID="71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b" Jul 08 02:25:39.297040 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.297018 2578 scope.go:117] "RemoveContainer" containerID="8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac" Jul 08 02:25:39.305440 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.305422 2578 scope.go:117] "RemoveContainer" containerID="71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b" Jul 08 02:25:39.305754 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:25:39.305732 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b\": container with ID starting with 71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b not found: ID does not exist" containerID="71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b" Jul 08 02:25:39.305830 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.305765 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b"} err="failed to get container status \"71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b\": rpc error: code = NotFound desc = could not find container \"71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b\": container with ID starting with 71b06550c630a6d2cfadc83f025787dd2679bc9eecf1354bf65d1b17cc98226b not found: ID does not exist" Jul 08 02:25:39.305830 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.305782 2578 scope.go:117] "RemoveContainer" containerID="8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac" Jul 08 02:25:39.306030 ip-10-0-134-84 kubenswrapper[2578]: E0708 02:25:39.306011 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac\": container with ID starting with 8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac not found: ID does not exist" containerID="8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac" Jul 08 02:25:39.306089 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.306042 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac"} err="failed to get container status \"8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac\": rpc error: code = NotFound desc = could not find container \"8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac\": container with ID starting with 8562feb68ad4a243b4ef6d7d79b71bfff4607fd98bdf5b3a50dc7d2938dbe4ac not found: ID does not exist" Jul 08 02:25:39.310318 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.310298 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8bbd955f5-c77pf_1c697a55-4ff5-42e1-9909-681e49531bab/copy-manifests/0.log" Jul 08 02:25:39.310911 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.310892 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m"] Jul 08 02:25:39.314101 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.314080 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b8b505-openshift-default-798484f767-24w8m"] Jul 08 02:25:39.317162 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.317146 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8bbd955f5-c77pf_1c697a55-4ff5-42e1-9909-681e49531bab/manager/0.log" Jul 08 02:25:39.906913 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:39.906884 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" path="/var/lib/kubelet/pods/96643d42-8630-4bd0-b5e9-a4ee97dfedc9/volumes" Jul 08 02:25:40.769303 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:40.769269 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575bbf8489-jq5t7_e311fc41-f7c2-4f54-8dee-c577ee2a6942/authorino/0.log" Jul 08 02:25:41.107148 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:41.107120 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vzw5k_5e6a88fd-1494-4658-bb29-58c2370ed927/kuadrant-console-plugin/0.log" Jul 08 02:25:41.471862 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:41.471790 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jvbzf_9dee3815-0b23-4a22-851d-10b7b1ea70aa/limitador/0.log" Jul 08 02:25:41.942176 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:41.942149 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4zhm6j_cebdefad-b4d7-4753-a0ea-b9e4e081db8e/istio-proxy/0.log" Jul 08 02:25:42.172197 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:42.172150 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bf889db5-hbprh_31844cd7-e4fc-4a12-9b76-88434d337770/kube-auth-proxy/0.log" Jul 08 02:25:43.706371 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:43.706338 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-v4zcj_40369d2a-3ead-4274-9f8d-e125abd67eaf/main/0.log" Jul 08 02:25:48.253708 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.253679 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qwgww/must-gather-r2b52"] Jul 08 02:25:48.254077 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.254014 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" Jul 08 02:25:48.254077 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.254025 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" Jul 08 02:25:48.254077 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.254038 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" Jul 08 02:25:48.254077 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.254043 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" Jul 08 02:25:48.254208 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.254090 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" Jul 08 02:25:48.254208 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.254099 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="96643d42-8630-4bd0-b5e9-a4ee97dfedc9" containerName="istio-proxy" Jul 08 02:25:48.254208 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.254107 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="528b0ce7-2144-409e-a9c7-0e595fd3b3c5" containerName="istio-proxy" Jul 08 02:25:48.258511 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.258470 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwgww/must-gather-r2b52" Jul 08 02:25:48.261011 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.260982 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qwgww\"/\"kube-root-ca.crt\"" Jul 08 02:25:48.261265 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.261250 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qwgww\"/\"openshift-service-ca.crt\"" Jul 08 02:25:48.261826 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.261810 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qwgww\"/\"default-dockercfg-pgwwr\"" Jul 08 02:25:48.264715 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.264694 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwgww/must-gather-r2b52"] Jul 08 02:25:48.321846 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.321812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13cce4bf-a78c-454d-a309-9589141ad6c1-must-gather-output\") pod \"must-gather-r2b52\" (UID: \"13cce4bf-a78c-454d-a309-9589141ad6c1\") " pod="openshift-must-gather-qwgww/must-gather-r2b52" Jul 08 02:25:48.322022 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.321860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrgt\" (UniqueName: \"kubernetes.io/projected/13cce4bf-a78c-454d-a309-9589141ad6c1-kube-api-access-bsrgt\") pod \"must-gather-r2b52\" (UID: \"13cce4bf-a78c-454d-a309-9589141ad6c1\") " pod="openshift-must-gather-qwgww/must-gather-r2b52" Jul 08 02:25:48.422953 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.422916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13cce4bf-a78c-454d-a309-9589141ad6c1-must-gather-output\") pod \"must-gather-r2b52\" (UID: \"13cce4bf-a78c-454d-a309-9589141ad6c1\") " pod="openshift-must-gather-qwgww/must-gather-r2b52" Jul 08 02:25:48.423130 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.422969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrgt\" (UniqueName: \"kubernetes.io/projected/13cce4bf-a78c-454d-a309-9589141ad6c1-kube-api-access-bsrgt\") pod \"must-gather-r2b52\" (UID: \"13cce4bf-a78c-454d-a309-9589141ad6c1\") " pod="openshift-must-gather-qwgww/must-gather-r2b52" Jul 08 02:25:48.423282 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.423262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13cce4bf-a78c-454d-a309-9589141ad6c1-must-gather-output\") pod \"must-gather-r2b52\" (UID: \"13cce4bf-a78c-454d-a309-9589141ad6c1\") " pod="openshift-must-gather-qwgww/must-gather-r2b52" Jul 08 02:25:48.431107 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.431080 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrgt\" (UniqueName: \"kubernetes.io/projected/13cce4bf-a78c-454d-a309-9589141ad6c1-kube-api-access-bsrgt\") pod \"must-gather-r2b52\" (UID: \"13cce4bf-a78c-454d-a309-9589141ad6c1\") " pod="openshift-must-gather-qwgww/must-gather-r2b52" Jul 08 02:25:48.568681 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.568573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwgww/must-gather-r2b52" Jul 08 02:25:48.699290 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.699263 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwgww/must-gather-r2b52"] Jul 08 02:25:48.701894 ip-10-0-134-84 kubenswrapper[2578]: W0708 02:25:48.701866 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13cce4bf_a78c_454d_a309_9589141ad6c1.slice/crio-910fdff3f6676ee03671e93d92fe0ebf02762b512f83aafeabd3a866cfe4377a WatchSource:0}: Error finding container 910fdff3f6676ee03671e93d92fe0ebf02762b512f83aafeabd3a866cfe4377a: Status 404 returned error can't find the container with id 910fdff3f6676ee03671e93d92fe0ebf02762b512f83aafeabd3a866cfe4377a Jul 08 02:25:48.703744 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:48.703727 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 02:25:49.334138 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:49.334102 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwgww/must-gather-r2b52" event={"ID":"13cce4bf-a78c-454d-a309-9589141ad6c1","Type":"ContainerStarted","Data":"910fdff3f6676ee03671e93d92fe0ebf02762b512f83aafeabd3a866cfe4377a"} Jul 08 02:25:50.343213 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:50.343176 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwgww/must-gather-r2b52" event={"ID":"13cce4bf-a78c-454d-a309-9589141ad6c1","Type":"ContainerStarted","Data":"1ec7614b8025d60f24a3b1cf7033161a07369dc27108f44913f224dd69b5fc99"} Jul 08 02:25:50.343728 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:50.343228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwgww/must-gather-r2b52" event={"ID":"13cce4bf-a78c-454d-a309-9589141ad6c1","Type":"ContainerStarted","Data":"f4258d478538e5133d8b73ede5dca9e54e3d9a1d55a0512b99fadbc2a26b7390"} Jul 08 02:25:50.359666 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:50.359601 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qwgww/must-gather-r2b52" podStartSLOduration=1.596312605 podStartE2EDuration="2.359575311s" totalImagesPullingTime="763.262706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:25:50.358188545 +0000 UTC m=+2581.020103082" watchObservedRunningTime="2026-07-08 02:25:50.359575311 +0000 UTC m=+2581.021489849" Jul 08 02:25:51.072848 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:51.072819 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fm4cg_1a6bb622-0bb3-4f7a-85bd-5714b6756d82/global-pull-secret-syncer/0.log" Jul 08 02:25:51.165519 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:51.165464 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l7qll_a5d6c6dc-5229-44fb-8744-715743709f96/konnectivity-agent/0.log" Jul 08 02:25:51.223353 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:51.223325 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-84.ec2.internal_e64f5c6be32d9511c588a4d2d5ea69a6/haproxy/0.log" Jul 08 02:25:56.917498 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:56.913739 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575bbf8489-jq5t7_e311fc41-f7c2-4f54-8dee-c577ee2a6942/authorino/0.log" Jul 08 02:25:56.993539 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:56.993511 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vzw5k_5e6a88fd-1494-4658-bb29-58c2370ed927/kuadrant-console-plugin/0.log" Jul 08 02:25:57.249348 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:57.249260 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jvbzf_9dee3815-0b23-4a22-851d-10b7b1ea70aa/limitador/0.log" Jul 08 02:25:59.136201 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.136163 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6mx4n_dfb84f22-be4a-453b-a595-c827e9bda0dd/kube-state-metrics/0.log" Jul 08 02:25:59.158419 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.158385 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6mx4n_dfb84f22-be4a-453b-a595-c827e9bda0dd/kube-rbac-proxy-main/0.log" Jul 08 02:25:59.179959 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.179933 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6mx4n_dfb84f22-be4a-453b-a595-c827e9bda0dd/kube-rbac-proxy-self/0.log" Jul 08 02:25:59.216868 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.216831 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6845648df-vv42s_63711f4d-76e8-4c1e-9666-5b7a5e112489/metrics-server/0.log" Jul 08 02:25:59.274997 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.274901 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kg52d_edafb75c-2df9-4915-91c2-3048cb8e7363/node-exporter/0.log" Jul 08 02:25:59.305674 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.305609 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kg52d_edafb75c-2df9-4915-91c2-3048cb8e7363/kube-rbac-proxy/0.log" Jul 08 02:25:59.338003 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.337977 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kg52d_edafb75c-2df9-4915-91c2-3048cb8e7363/init-textfile/0.log" Jul 08 02:25:59.544702 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.544611 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-lffd9_5c0a460b-0fa2-4119-8e7b-c2cd4d792964/kube-rbac-proxy-main/0.log" Jul 08 02:25:59.576546 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.576517 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-lffd9_5c0a460b-0fa2-4119-8e7b-c2cd4d792964/kube-rbac-proxy-self/0.log" Jul 08 02:25:59.604930 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.604899 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-lffd9_5c0a460b-0fa2-4119-8e7b-c2cd4d792964/openshift-state-metrics/0.log" Jul 08 02:25:59.642389 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.642356 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93f0548d-c223-4f5a-8656-691aa551e5a1/prometheus/0.log" Jul 08 02:25:59.664320 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.664292 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93f0548d-c223-4f5a-8656-691aa551e5a1/config-reloader/0.log" Jul 08 02:25:59.689212 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.689179 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93f0548d-c223-4f5a-8656-691aa551e5a1/thanos-sidecar/0.log" Jul 08 02:25:59.711055 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.711020 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93f0548d-c223-4f5a-8656-691aa551e5a1/kube-rbac-proxy-web/0.log" Jul 08 02:25:59.732757 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.732730 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93f0548d-c223-4f5a-8656-691aa551e5a1/kube-rbac-proxy/0.log" Jul 08 02:25:59.761039 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.761003 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93f0548d-c223-4f5a-8656-691aa551e5a1/kube-rbac-proxy-thanos/0.log" Jul 08 02:25:59.782662 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.782634 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93f0548d-c223-4f5a-8656-691aa551e5a1/init-config-reloader/0.log" Jul 08 02:25:59.857619 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.857588 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-xkqr2_499318ab-de32-4681-9c47-622bf9517e65/prometheus-operator-admission-webhook/0.log" Jul 08 02:25:59.987855 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:25:59.987823 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b578d7c8-kf8jr_15c69306-2ece-463f-9af8-29ee52213869/thanos-query/0.log" Jul 08 02:26:00.009070 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:00.009044 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b578d7c8-kf8jr_15c69306-2ece-463f-9af8-29ee52213869/kube-rbac-proxy-web/0.log" Jul 08 02:26:00.028988 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:00.028963 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b578d7c8-kf8jr_15c69306-2ece-463f-9af8-29ee52213869/kube-rbac-proxy/0.log" Jul 08 02:26:00.063064 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:00.063033 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b578d7c8-kf8jr_15c69306-2ece-463f-9af8-29ee52213869/prom-label-proxy/0.log" Jul 08 02:26:00.085304 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:00.085280 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b578d7c8-kf8jr_15c69306-2ece-463f-9af8-29ee52213869/kube-rbac-proxy-rules/0.log" Jul 08 02:26:00.108023 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:00.107917 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9b578d7c8-kf8jr_15c69306-2ece-463f-9af8-29ee52213869/kube-rbac-proxy-metrics/0.log" Jul 08 02:26:01.544306 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.544264 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647"] Jul 08 02:26:01.552525 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.552496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.555552 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.555265 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647"] Jul 08 02:26:01.665827 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.665773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-sys\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.666024 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.665857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-podres\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.666024 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.665922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-lib-modules\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.666024 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.665963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjgfl\" (UniqueName: \"kubernetes.io/projected/b3fe01e6-0870-4299-a23e-095b3bc5aa14-kube-api-access-xjgfl\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.666216 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.666103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-proc\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.767261 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.767227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-sys\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.767444 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.767282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-podres\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.767444 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.767325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-lib-modules\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.767444 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.767350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjgfl\" (UniqueName: \"kubernetes.io/projected/b3fe01e6-0870-4299-a23e-095b3bc5aa14-kube-api-access-xjgfl\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.767444 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.767386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-sys\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.767697 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.767514 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-podres\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.767697 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.767532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-proc\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.767697 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.767545 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-lib-modules\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.767697 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.767599 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3fe01e6-0870-4299-a23e-095b3bc5aa14-proc\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.776783 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.776740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjgfl\" (UniqueName: \"kubernetes.io/projected/b3fe01e6-0870-4299-a23e-095b3bc5aa14-kube-api-access-xjgfl\") pod \"perf-node-gather-daemonset-fs647\" (UID: \"b3fe01e6-0870-4299-a23e-095b3bc5aa14\") " pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:01.865623 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:01.865426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:02.060925 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:02.060697 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647"] Jul 08 02:26:02.438362 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:02.438264 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" event={"ID":"b3fe01e6-0870-4299-a23e-095b3bc5aa14","Type":"ContainerStarted","Data":"d60cdfcd853c656b424c373574a2e4848a91ae218d2d1dd87d6d25bb7885f985"} Jul 08 02:26:02.438362 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:02.438326 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:02.438362 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:02.438344 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" event={"ID":"b3fe01e6-0870-4299-a23e-095b3bc5aa14","Type":"ContainerStarted","Data":"bdb293e2145fd4e8f0937507ccb36b40625b19fb5b0bf89d6a2f5673c479218f"} Jul 08 02:26:02.464472 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:02.464424 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" podStartSLOduration=1.464410402 podStartE2EDuration="1.464410402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:26:02.461812356 +0000 UTC m=+2593.123726884" watchObservedRunningTime="2026-07-08 02:26:02.464410402 +0000 UTC m=+2593.126324939" Jul 08 02:26:02.481371 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:02.481347 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58f6c9449-d479d_f06e966a-cd7f-43b0-80d0-ba771ba448fb/console/0.log" Jul 08 02:26:04.184126 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:04.184098 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wdpsc_ee5e9319-202e-4401-8772-d2fafe6eb896/dns/0.log" Jul 08 02:26:04.207372 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:04.207345 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wdpsc_ee5e9319-202e-4401-8772-d2fafe6eb896/kube-rbac-proxy/0.log" Jul 08 02:26:04.230985 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:04.230960 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-66zvg_392a0884-e501-4432-870e-bd698aaa7616/dns-node-resolver/0.log" Jul 08 02:26:04.847611 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:04.847581 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54f2j_a542d4a8-3f11-4d97-b235-71de248f8a72/node-ca/0.log" Jul 08 02:26:05.791168 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:05.791139 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4zhm6j_cebdefad-b4d7-4753-a0ea-b9e4e081db8e/istio-proxy/0.log" Jul 08 02:26:05.923231 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:05.923203 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bf889db5-hbprh_31844cd7-e4fc-4a12-9b76-88434d337770/kube-auth-proxy/0.log" Jul 08 02:26:06.865009 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:06.864979 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tbvrq_5eb9e5b4-0014-49e9-87e6-e6aea2920399/serve-healthcheck-canary/0.log" Jul 08 02:26:07.542570 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:07.542537 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-47hv5_de36bd07-f274-429f-b46d-05bb02465482/kube-rbac-proxy/0.log" Jul 08 02:26:07.595727 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:07.595700 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-47hv5_de36bd07-f274-429f-b46d-05bb02465482/exporter/0.log" Jul 08 02:26:07.646057 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:07.646029 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-47hv5_de36bd07-f274-429f-b46d-05bb02465482/extractor/0.log" Jul 08 02:26:08.456628 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:08.456595 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qwgww/perf-node-gather-daemonset-fs647" Jul 08 02:26:10.032722 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:10.032637 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-m2bv4_bcf78b36-1ffe-4646-b9b3-c3ecabf5eecf/manager/0.log" Jul 08 02:26:10.470301 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:10.470268 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8bbd955f5-c77pf_1c697a55-4ff5-42e1-9909-681e49531bab/manager/0.log" Jul 08 02:26:10.492500 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:10.492452 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8bbd955f5-c77pf_1c697a55-4ff5-42e1-9909-681e49531bab/copy-manifests/0.log" Jul 08 02:26:11.576828 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:11.576795 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4zhm6j_cebdefad-b4d7-4753-a0ea-b9e4e081db8e/istio-proxy/0.log" Jul 08 02:26:11.697959 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:11.697928 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bf889db5-hbprh_31844cd7-e4fc-4a12-9b76-88434d337770/kube-auth-proxy/0.log" Jul 08 02:26:11.884819 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:11.884787 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7769fc489f-xlh57_c2ba6d26-f752-48ee-b4ba-bffa7df36502/manager/0.log" Jul 08 02:26:13.012677 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:13.012600 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4zhm6j_cebdefad-b4d7-4753-a0ea-b9e4e081db8e/istio-proxy/0.log" Jul 08 02:26:13.137189 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:13.137152 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bf889db5-hbprh_31844cd7-e4fc-4a12-9b76-88434d337770/kube-auth-proxy/0.log" Jul 08 02:26:18.426912 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.426880 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mn7c_529292c4-1ccf-4ebc-a014-af5094659f0c/kube-multus-additional-cni-plugins/0.log" Jul 08 02:26:18.449444 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.449415 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mn7c_529292c4-1ccf-4ebc-a014-af5094659f0c/egress-router-binary-copy/0.log" Jul 08 02:26:18.469703 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.469679 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mn7c_529292c4-1ccf-4ebc-a014-af5094659f0c/cni-plugins/0.log" Jul 08 02:26:18.489466 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.489441 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mn7c_529292c4-1ccf-4ebc-a014-af5094659f0c/bond-cni-plugin/0.log" Jul 08 02:26:18.510021 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.509996 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mn7c_529292c4-1ccf-4ebc-a014-af5094659f0c/routeoverride-cni/0.log" Jul 08 02:26:18.531306 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.531278 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mn7c_529292c4-1ccf-4ebc-a014-af5094659f0c/whereabouts-cni-bincopy/0.log" Jul 08 02:26:18.550962 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.550888 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mn7c_529292c4-1ccf-4ebc-a014-af5094659f0c/whereabouts-cni/0.log" Jul 08 02:26:18.731677 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.731646 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m5xj7_061de9e3-7d37-4c96-aece-75722b0c41c8/kube-multus/0.log" Jul 08 02:26:18.861196 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.861157 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-852qz_85d121a8-c00e-4baf-a410-40d9ac037d1c/network-metrics-daemon/0.log" Jul 08 02:26:18.879429 ip-10-0-134-84 kubenswrapper[2578]: I0708 02:26:18.879403 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-852qz_85d121a8-c00e-4baf-a410-40d9ac037d1c/kube-rbac-proxy/0.log"