Jul 06 19:17:59.331493 ip-10-0-131-211 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 19:17:59.331508 ip-10-0-131-211 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 19:17:59.331515 ip-10-0-131-211 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 19:17:59.331744 ip-10-0-131-211 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 19:18:09.472796 ip-10-0-131-211 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 19:18:09.472814 ip-10-0-131-211 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0ff9f78d3cd046a1bb92f738f85329af -- Jul 06 19:20:12.832554 ip-10-0-131-211 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:20:13.200263 ip-10-0-131-211 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:20:13.200263 ip-10-0-131-211 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:20:13.200263 ip-10-0-131-211 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:20:13.200263 ip-10-0-131-211 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:20:13.200263 ip-10-0-131-211 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:20:13.201175 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.201090 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:20:13.205937 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205922 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:20:13.205937 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205936 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205940 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205943 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205946 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205948 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205951 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205953 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205956 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205958 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205961 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205963 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205966 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205969 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205971 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205976 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205979 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205982 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205985 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205988 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:20:13.205997 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205991 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205994 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.205997 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206000 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206012 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206015 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206017 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206020 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206023 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206025 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206028 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206030 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206032 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206035 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206037 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206040 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206042 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206044 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206047 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206049 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:20:13.206474 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206052 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206054 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206057 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206060 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206062 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206065 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206067 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206070 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206074 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206078 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206081 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206083 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206085 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206088 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206091 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206093 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206095 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206098 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206100 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206103 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:20:13.206990 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206105 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206108 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206110 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206112 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206115 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206117 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206120 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206122 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206125 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206127 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206130 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206133 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206135 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206138 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206140 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206143 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206146 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206148 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206150 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206154 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:20:13.207546 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206157 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206160 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206162 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206165 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206168 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206172 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206175 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206178 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206181 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206183 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206186 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206189 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206191 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206194 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206196 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206198 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206201 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206203 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206206 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206935 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:20:13.208022 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206942 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206945 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206947 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206950 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206958 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206961 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206964 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206967 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206969 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206972 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206975 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206977 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206980 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206982 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206985 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206987 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206989 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206993 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206996 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:20:13.208514 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.206998 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207001 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207004 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207006 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207009 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207011 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207013 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207016 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207018 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207021 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207023 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207026 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207028 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207031 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207033 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207036 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207038 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207040 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207044 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207046 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:20:13.208971 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207049 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207054 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207057 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207060 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207063 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207066 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207080 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207082 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207085 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207087 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207090 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207093 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207095 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207097 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207100 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207103 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207105 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207107 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207110 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207112 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:20:13.209473 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207114 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207117 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207119 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207121 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207124 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207128 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207132 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207135 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207137 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207140 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207142 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207146 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207148 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207151 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207154 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207156 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207159 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207161 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207164 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207167 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:20:13.209959 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207169 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207172 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207174 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207177 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207179 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207182 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207185 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207188 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207190 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207193 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207195 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207198 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207200 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207202 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207205 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207210 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207213 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207216 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207219 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:20:13.210451 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207612 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207617 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207620 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207624 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207629 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207632 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207634 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207637 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207639 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207642 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207644 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207647 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207649 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207652 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207655 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207657 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207660 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207663 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207666 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207668 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:20:13.210899 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207670 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207673 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207675 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207678 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207680 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207683 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207685 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207687 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207690 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207692 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207694 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207697 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207699 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207702 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207704 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207706 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207709 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207712 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207714 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207717 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:20:13.211419 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207719 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207722 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207724 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207726 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207729 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207731 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207733 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207736 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207740 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207743 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207745 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207750 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207753 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207756 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207759 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207761 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207764 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207766 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207769 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:20:13.211887 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207771 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207773 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207776 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207778 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207781 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207783 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207786 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207788 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207791 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207794 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207796 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207799 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207802 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207804 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207807 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207810 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207812 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207815 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207817 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207819 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:20:13.212410 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207824 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207827 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207829 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207832 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207834 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207836 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207839 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207841 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207844 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207846 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207848 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207851 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207853 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207855 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207858 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207860 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207862 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207865 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207867 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.207870 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:20:13.212884 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207940 2571 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207947 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207960 2571 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207965 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207971 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207974 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207978 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207982 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207985 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207988 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207991 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207994 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.207998 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208001 2571 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208004 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208007 2571 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208009 2571 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208012 2571 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208017 2571 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208019 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208022 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208025 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208028 2571 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208032 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:20:13.213378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208035 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208038 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208041 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208044 2571 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208047 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208050 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208053 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208055 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208059 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208062 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208067 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208069 2571 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208073 2571 flags.go:64] FLAG: --enable-server="true" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208076 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208080 2571 flags.go:64] FLAG: --event-burst="100" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208083 2571 flags.go:64] FLAG: --event-qps="50" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208085 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208088 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208091 2571 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208094 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208097 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208100 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208103 2571 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208105 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208108 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:20:13.213970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208111 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208114 2571 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208116 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208120 2571 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208122 2571 flags.go:64] FLAG: --feature-gates="" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208126 2571 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208128 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208131 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208134 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208137 2571 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208139 2571 flags.go:64] FLAG: --help="false" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208142 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208145 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208148 2571 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208151 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208154 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208157 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208162 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208165 2571 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208167 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208170 2571 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208173 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208176 2571 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208179 2571 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:20:13.214592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208182 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208184 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208187 2571 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208190 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208193 2571 flags.go:64] FLAG: --lock-file="" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208195 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208198 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208201 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208206 2571 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208209 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208211 2571 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208214 2571 flags.go:64] FLAG: --logging-format="text" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208217 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208220 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208222 2571 flags.go:64] FLAG: --manifest-url="" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208225 2571 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208228 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208231 2571 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208235 2571 flags.go:64] FLAG: --max-pods="110" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208238 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208241 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208243 2571 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208246 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208249 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208252 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:20:13.215185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208256 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208262 2571 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208265 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208281 2571 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208285 2571 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208288 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208293 2571 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208296 2571 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208299 2571 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208302 2571 flags.go:64] FLAG: --port="10250" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208305 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208308 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01a63d9b7a0e015c1" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208311 2571 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208314 2571 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208317 2571 flags.go:64] FLAG: --register-node="true" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208319 2571 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208323 2571 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208325 2571 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208328 2571 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208331 2571 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208334 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208337 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208340 2571 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208342 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208345 2571 flags.go:64] FLAG: --runonce="false" Jul 06 19:20:13.215803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208348 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208351 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208354 2571 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208357 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208359 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208362 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208365 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208368 2571 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208371 2571 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208374 2571 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208377 2571 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208380 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208383 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208386 2571 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208389 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208394 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208397 2571 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208400 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208404 2571 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208406 2571 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208409 2571 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208411 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208414 2571 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208417 2571 flags.go:64] FLAG: --v="2" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208421 2571 flags.go:64] FLAG: --version="false" Jul 06 19:20:13.216424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208425 2571 flags.go:64] FLAG: --vmodule="" Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208428 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.208431 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208540 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208544 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208546 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208550 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208552 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208555 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208557 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208560 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208562 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208566 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208569 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208572 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208577 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208580 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208582 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208585 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208588 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:20:13.217013 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208591 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208595 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208599 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208604 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208606 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208609 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208611 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208614 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208617 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208619 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208622 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208625 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208628 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208630 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208633 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208636 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208638 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208641 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208644 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:20:13.217540 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208646 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208649 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208651 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208655 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208658 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208661 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208663 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208665 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208669 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208671 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208674 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208676 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208679 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208681 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208684 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208687 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208690 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208693 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208695 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:20:13.217983 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208697 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208700 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208702 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208704 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208707 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208710 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208712 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208714 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208717 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208719 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208722 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208724 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208726 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208729 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208731 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208734 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208736 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208739 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208741 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208743 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:20:13.218478 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208746 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208750 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208752 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208755 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208757 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208760 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208762 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208765 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208767 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208771 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208774 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208776 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208779 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208781 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208784 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208786 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208788 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208791 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208793 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208795 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:20:13.218968 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208798 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208800 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208803 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.208805 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.209548 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.216678 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.216695 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.217619 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.217626 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217686 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217690 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217694 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217697 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217699 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217702 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:20:13.219463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217704 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217707 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217710 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217712 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217715 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217718 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217720 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217723 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217725 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217728 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217731 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217733 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217735 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217738 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217740 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217743 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217745 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217748 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217750 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217753 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:20:13.219830 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217755 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217758 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217760 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217772 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217776 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217780 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217783 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217786 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217788 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217791 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217793 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217796 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217798 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217801 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217803 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217806 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217808 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217811 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217813 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217816 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:20:13.220310 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217819 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217821 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217824 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217826 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217829 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217831 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217834 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217836 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217838 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217841 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217843 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217846 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217850 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217853 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217856 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217859 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217870 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217874 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217876 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217879 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:20:13.220783 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217881 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217884 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217886 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217889 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217892 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217894 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217897 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217899 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217902 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217904 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217907 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217910 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217913 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217916 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217919 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217921 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217924 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217926 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217929 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:20:13.221261 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217931 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217933 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217936 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217938 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217941 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217943 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217946 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217948 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217951 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217954 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217966 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217969 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217971 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.217973 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.217979 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:20:13.221717 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218128 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218133 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218135 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218138 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218141 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218143 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218146 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218149 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218151 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218154 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218156 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218159 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218161 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218163 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218166 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218169 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218171 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218174 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218176 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218179 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:20:13.222081 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218181 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218183 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218186 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218189 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218192 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218195 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218197 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218200 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218208 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218210 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218212 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218215 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218217 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218220 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218223 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218227 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218229 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218231 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218234 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:20:13.222568 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218236 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218240 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218243 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218246 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218249 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218251 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218254 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218257 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218259 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218262 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218264 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218266 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218288 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218292 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218295 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218298 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218301 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218304 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218306 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:20:13.223020 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218310 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218313 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218315 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218324 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218327 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218329 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218332 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218334 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218336 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218339 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218341 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218344 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218346 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218349 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218351 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218354 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218356 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218358 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218361 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218363 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:20:13.223596 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218365 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218368 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218371 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218373 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218376 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218378 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218380 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218383 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218385 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218388 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218390 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218393 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218395 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218397 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218400 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218402 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218410 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218412 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218415 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218417 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:20:13.224069 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:13.218420 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:20:13.224549 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.218424 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:20:13.224549 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.218539 2571 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:20:13.224549 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.220940 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:20:13.224549 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.222304 2571 server.go:1019] "Starting client certificate rotation" Jul 06 19:20:13.224549 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.222411 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:20:13.224549 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.222454 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:20:13.225112 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.225096 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:20:13.230850 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.230826 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:20:13.239027 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.239012 2571 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:20:13.241318 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.241305 2571 log.go:25] "Validated CRI v1 image API" Jul 06 19:20:13.242817 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.242787 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:20:13.246106 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.246090 2571 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c1c4ce62-653a-4db5-b344-9ef4571c8959:/dev/nvme0n1p3 f9129e0b-f203-49d3-ad18-4e115d57f7e4:/dev/nvme0n1p4] Jul 06 19:20:13.246171 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.246107 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:20:13.250288 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.250180 2571 manager.go:216] Machine: {Timestamp:2026-07-06 19:20:13.249729732 +0000 UTC m=+0.310076649 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102787 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27b7e3033da5f5b2bb3f6c21271af0 SystemUUID:ec27b7e3-033d-a5f5-b2bb-3f6c21271af0 BootID:0ff9f78d-3cd0-46a1-bb92-f738f85329af Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:40:3b:fa:4f:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:40:3b:fa:4f:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:fb:20:9d:91:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:20:13.250357 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.250288 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:20:13.250399 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.250382 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:20:13.252159 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.252133 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:20:13.252307 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.252163 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-211.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:20:13.252936 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.252926 2571 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:20:13.252969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.252941 2571 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:20:13.252969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.252952 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:20:13.253509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.253498 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:20:13.254646 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.254631 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:20:13.254982 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.254972 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:20:13.258122 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.258110 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:20:13.258376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.258366 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:20:13.258407 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.258399 2571 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:20:13.258433 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.258409 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:20:13.258433 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.258424 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:20:13.258433 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.258432 2571 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:20:13.258525 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.258441 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:20:13.258784 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.258770 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:20:13.258838 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.258789 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:20:13.260831 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.260816 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:20:13.263105 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.263088 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:20:13.263153 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.263122 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:20:13.264217 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264206 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:20:13.264261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264221 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:20:13.264261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264227 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:20:13.264261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264232 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:20:13.264261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264240 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:20:13.264261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264246 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:20:13.264261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264254 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:20:13.264261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264259 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:20:13.264261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264264 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:20:13.264476 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264285 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:20:13.264476 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264294 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:20:13.264476 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.264299 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:20:13.265757 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.265732 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:20:13.266661 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.266644 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:20:13.270097 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.270080 2571 server.go:1268] "Started kubelet" Jul 06 19:20:13.270219 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.270172 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:20:13.270219 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.270184 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:20:13.270335 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.270245 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:20:13.270986 ip-10-0-131-211 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:20:13.272491 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.272267 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:20:13.272571 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.272363 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:20:13.272720 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.272703 2571 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:20:13.272870 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.272846 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-211.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:20:13.272917 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.272868 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:20:13.275221 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.275204 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:20:13.277654 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.276827 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-211.ec2.internal.18bfc9939b9bb56c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-211.ec2.internal,UID:ip-10-0-131-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-211.ec2.internal,},FirstTimestamp:2026-07-06 19:20:13.270054252 +0000 UTC m=+0.330401169,LastTimestamp:2026-07-06 19:20:13.270054252 +0000 UTC m=+0.330401169,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-211.ec2.internal,}" Jul 06 19:20:13.279686 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.279666 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:20:13.279686 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.279682 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:20:13.280096 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.280060 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:13.280408 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280384 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:20:13.280481 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280449 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:20:13.280481 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280463 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:20:13.280753 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280539 2571 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:20:13.280753 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280542 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:20:13.280753 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280548 2571 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:20:13.280753 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280553 2571 factory.go:55] Registering systemd factory Jul 06 19:20:13.280753 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280559 2571 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:20:13.280964 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280804 2571 factory.go:153] Registering CRI-O factory Jul 06 19:20:13.280964 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280814 2571 factory.go:223] Registration of the crio container factory successfully Jul 06 19:20:13.280964 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280841 2571 factory.go:103] Registering Raw factory Jul 06 19:20:13.280964 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.280853 2571 manager.go:1196] Started watching for new ooms in manager Jul 06 19:20:13.281240 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.281225 2571 manager.go:349] Starting recovery of all containers Jul 06 19:20:13.283318 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.283295 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jwf8c" Jul 06 19:20:13.286815 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.286776 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:20:13.286930 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.286908 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:20:13.290808 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.289778 2571 manager.go:354] Recovery completed Jul 06 19:20:13.291750 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.291725 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jwf8c" Jul 06 19:20:13.296683 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.296669 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:20:13.299319 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.299299 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:20:13.299399 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.299335 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:20:13.299399 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.299349 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:20:13.299897 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.299885 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:20:13.299951 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.299884 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:20:13.299991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.299951 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:20:13.299991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.299968 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:20:13.300528 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.300515 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:20:13.300575 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.300541 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:20:13.300575 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.300551 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:20:13.301464 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.301430 2571 policy_none.go:49] "None policy: Start" Jul 06 19:20:13.301496 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.301467 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:20:13.301496 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.301478 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:20:13.301698 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.301618 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-211.ec2.internal.18bfc9939d5a42a7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-211.ec2.internal,UID:ip-10-0-131-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-211.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-211.ec2.internal,},FirstTimestamp:2026-07-06 19:20:13.299319463 +0000 UTC m=+0.359666578,LastTimestamp:2026-07-06 19:20:13.299319463 +0000 UTC m=+0.359666578,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-211.ec2.internal,}" Jul 06 19:20:13.302928 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.302916 2571 policy_none.go:47] "Start" Jul 06 19:20:13.338830 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.338816 2571 manager.go:337] "Starting Device Plugin manager" Jul 06 19:20:13.338914 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.338855 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:20:13.338914 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.338864 2571 server.go:87] "Starting device plugin registration server" Jul 06 19:20:13.339108 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.339095 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:20:13.339144 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.339112 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:20:13.339238 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.339214 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:20:13.339332 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.339317 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:20:13.339332 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.339330 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:20:13.339770 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.339747 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:20:13.339839 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.339802 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:13.439734 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.439713 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:20:13.440726 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.440669 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:20:13.440726 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.440698 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:20:13.440726 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.440712 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:20:13.440835 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.440740 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.448773 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.448759 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.448815 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.448785 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-211.ec2.internal\": node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:13.453257 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.453207 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:20:13.454648 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.454626 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:20:13.454713 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.454671 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:20:13.454713 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.454690 2571 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:20:13.454810 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.454718 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:20:13.456618 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.456598 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:20:13.464151 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.464129 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:13.555318 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.555296 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal"] Jul 06 19:20:13.555376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.555359 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:20:13.556746 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.556731 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:20:13.556800 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.556758 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:20:13.556800 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.556768 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:20:13.558052 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.558040 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:20:13.558157 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.558143 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.558206 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.558173 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:20:13.558632 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.558616 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:20:13.558709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.558645 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:20:13.558709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.558652 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:20:13.558709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.558659 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:20:13.558709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.558672 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:20:13.558709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.558683 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:20:13.560345 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.560330 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.560422 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.560355 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:20:13.560939 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.560924 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:20:13.561015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.560954 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:20:13.561015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.560967 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:20:13.564218 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.564204 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:13.582751 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.582729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee9ea7bd3edf3de498f018682e7d0a02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal\" (UID: \"ee9ea7bd3edf3de498f018682e7d0a02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.582835 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.582760 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee9ea7bd3edf3de498f018682e7d0a02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal\" (UID: \"ee9ea7bd3edf3de498f018682e7d0a02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.582835 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.582775 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-211.ec2.internal\" not found" node="ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.582835 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.582777 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc1d303b9ebb2e2da2b7750642266d77-config\") pod \"kube-apiserver-proxy-ip-10-0-131-211.ec2.internal\" (UID: \"cc1d303b9ebb2e2da2b7750642266d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.587175 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.587160 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-211.ec2.internal\" not found" node="ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.665136 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.665115 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:13.682902 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.682882 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee9ea7bd3edf3de498f018682e7d0a02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal\" (UID: \"ee9ea7bd3edf3de498f018682e7d0a02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.682967 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.682941 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee9ea7bd3edf3de498f018682e7d0a02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal\" (UID: \"ee9ea7bd3edf3de498f018682e7d0a02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.683006 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.682983 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee9ea7bd3edf3de498f018682e7d0a02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal\" (UID: \"ee9ea7bd3edf3de498f018682e7d0a02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.683039 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.683006 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee9ea7bd3edf3de498f018682e7d0a02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal\" (UID: \"ee9ea7bd3edf3de498f018682e7d0a02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.683039 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.683019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc1d303b9ebb2e2da2b7750642266d77-config\") pod \"kube-apiserver-proxy-ip-10-0-131-211.ec2.internal\" (UID: \"cc1d303b9ebb2e2da2b7750642266d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.683109 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.683048 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc1d303b9ebb2e2da2b7750642266d77-config\") pod \"kube-apiserver-proxy-ip-10-0-131-211.ec2.internal\" (UID: \"cc1d303b9ebb2e2da2b7750642266d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.765706 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.765649 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:13.866130 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.866103 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:13.887627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.887601 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.892081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:13.891794 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal" Jul 06 19:20:13.967034 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:13.967012 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:14.067623 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:14.067571 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:14.168062 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:14.168037 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:14.221557 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.221533 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:20:14.222059 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.221654 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:20:14.269066 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:14.269042 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:14.295897 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.295857 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:15:13 +0000 UTC" deadline="2028-02-10 11:06:55.343779713 +0000 UTC" Jul 06 19:20:14.295897 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.295895 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14007h46m41.047889139s" Jul 06 19:20:14.369428 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:14.369402 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-211.ec2.internal\" not found" Jul 06 19:20:14.378384 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:14.378336 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1d303b9ebb2e2da2b7750642266d77.slice/crio-e7a6b5834bd010cdeba31ad7e6bca48997413abf4ff79ba05ce82bcab46a01cf WatchSource:0}: Error finding container e7a6b5834bd010cdeba31ad7e6bca48997413abf4ff79ba05ce82bcab46a01cf: Status 404 returned error can't find the container with id e7a6b5834bd010cdeba31ad7e6bca48997413abf4ff79ba05ce82bcab46a01cf Jul 06 19:20:14.378626 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:14.378602 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9ea7bd3edf3de498f018682e7d0a02.slice/crio-9bf505429b22c588b90dd9df42edc60053ddea30f6883a4dfee38e3d1343cdb5 WatchSource:0}: Error finding container 9bf505429b22c588b90dd9df42edc60053ddea30f6883a4dfee38e3d1343cdb5: Status 404 returned error can't find the container with id 9bf505429b22c588b90dd9df42edc60053ddea30f6883a4dfee38e3d1343cdb5 Jul 06 19:20:14.382141 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.382126 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:20:14.385643 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.385622 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:20:14.423527 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.423502 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:20:14.457677 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.457632 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" event={"ID":"ee9ea7bd3edf3de498f018682e7d0a02","Type":"ContainerStarted","Data":"9bf505429b22c588b90dd9df42edc60053ddea30f6883a4dfee38e3d1343cdb5"} Jul 06 19:20:14.458759 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.458737 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal" event={"ID":"cc1d303b9ebb2e2da2b7750642266d77","Type":"ContainerStarted","Data":"e7a6b5834bd010cdeba31ad7e6bca48997413abf4ff79ba05ce82bcab46a01cf"} Jul 06 19:20:14.480057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.480034 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" Jul 06 19:20:14.492163 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.492147 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:20:14.493222 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.493209 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal" Jul 06 19:20:14.501930 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.501914 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:20:14.569975 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:14.569956 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:20:15.259926 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.259889 2571 apiserver.go:52] "Watching apiserver" Jul 06 19:20:15.266586 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.266386 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:20:15.266871 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.266843 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-n57p8","openshift-network-diagnostics/network-check-target-h4s4w","openshift-network-operator/iptables-alerter-cs5dn","openshift-ovn-kubernetes/ovnkube-node-9xc2t","kube-system/konnectivity-agent-nfhbg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh","openshift-cluster-node-tuning-operator/tuned-mkxfr","openshift-dns/node-resolver-cb5fx","openshift-image-registry/node-ca-tbczb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal","openshift-multus/multus-additional-cni-plugins-hd6m4","openshift-multus/network-metrics-daemon-gdvls","kube-system/global-pull-secret-syncer-8rchd","kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal"] Jul 06 19:20:15.268957 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.268846 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.270166 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.270140 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.270275 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.270219 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:15.271747 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.271243 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.271747 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.271370 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:20:15.271747 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.271527 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wbqhz\"" Jul 06 19:20:15.271747 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.271608 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:20:15.273089 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.273068 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.274126 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.274107 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:20:15.274355 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.274320 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:20:15.274527 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.274509 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:20:15.274809 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.274784 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hsvq2\"" Jul 06 19:20:15.275923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.275582 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:15.275923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.275724 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:20:15.275923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.275918 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:20:15.276798 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.276448 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:20:15.276798 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.276645 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:20:15.276798 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.276698 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:20:15.276798 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.276774 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:20:15.277095 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.277056 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n6tvk\"" Jul 06 19:20:15.277839 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.277786 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:20:15.278306 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.278234 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.279588 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.279357 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5ccmb\"" Jul 06 19:20:15.279588 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.279450 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:20:15.279720 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.279608 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.279961 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.279937 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:20:15.280283 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.280248 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.280368 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.280354 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:20:15.282767 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.282743 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gfbt5\"" Jul 06 19:20:15.283400 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.283384 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vstc2\"" Jul 06 19:20:15.283827 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.283798 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:20:15.284002 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.283987 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:20:15.284182 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.284170 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:20:15.284514 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.284495 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:20:15.284624 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.284604 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:20:15.284784 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.284767 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:20:15.285055 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.285034 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:20:15.285134 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.285102 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:20:15.285210 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.285039 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mr77v\"" Jul 06 19:20:15.285391 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.285373 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:20:15.287759 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.287740 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:15.287853 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.287819 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:15.289759 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.289735 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.291247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.291223 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.292219 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292195 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xbppp\"" Jul 06 19:20:15.292454 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292433 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:15.292454 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292448 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:20:15.292539 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.292502 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:15.292692 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292672 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-kubelet\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.292692 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292684 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:20:15.292849 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292707 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-var-lib-kubelet\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.292849 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292736 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-run-ovn\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.292849 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292760 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-cni-binary-copy\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.292849 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292785 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-conf-dir\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.292849 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpxbh\" (UniqueName: \"kubernetes.io/projected/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-kube-api-access-kpxbh\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.293093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292859 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-run-openvswitch\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.293093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292919 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-run-ovn-kubernetes\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.293093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292945 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44d4a5a8-5d12-483d-939c-5e40d5d3658b-kubelet-config\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.293093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.292975 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-cni-dir\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.293093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293000 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-socket-dir-parent\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.293093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293002 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:20:15.293093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293023 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-log-socket\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.293093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293047 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44d4a5a8-5d12-483d-939c-5e40d5d3658b-dbus\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293092 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7687a7fd-1cbf-4521-a462-e976d785f5e5-serviceca\") pod \"node-ca-tbczb\" (UID: \"7687a7fd-1cbf-4521-a462-e976d785f5e5\") " pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293123 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-run-netns\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293150 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293196 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293213 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7687a7fd-1cbf-4521-a462-e976d785f5e5-host\") pod \"node-ca-tbczb\" (UID: \"7687a7fd-1cbf-4521-a462-e976d785f5e5\") " pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293232 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-system-cni-dir\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293248 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-hostroot\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293262 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-sysctl-d\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293277 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-run-systemd\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293313 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-etc-openvswitch\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293348 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15fee54d-02d3-4d29-a6de-d0c58007438d-ovnkube-config\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d99aa6b-369e-4f8e-984f-1b3097abfbcf-agent-certs\") pod \"konnectivity-agent-nfhbg\" (UID: \"5d99aa6b-369e-4f8e-984f-1b3097abfbcf\") " pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293394 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-run-netns\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293419 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-var-lib-cni-bin\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-run\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.293502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293490 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-sys\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293515 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d99aa6b-369e-4f8e-984f-1b3097abfbcf-konnectivity-ca\") pod \"konnectivity-agent-nfhbg\" (UID: \"5d99aa6b-369e-4f8e-984f-1b3097abfbcf\") " pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-systemd\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftwj\" (UniqueName: \"kubernetes.io/projected/7687a7fd-1cbf-4521-a462-e976d785f5e5-kube-api-access-zftwj\") pod \"node-ca-tbczb\" (UID: \"7687a7fd-1cbf-4521-a462-e976d785f5e5\") " pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293590 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-run-multus-certs\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293612 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-etc-kubernetes\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293628 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-kubernetes\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293642 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-sysctl-conf\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293663 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-cni-netd\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293691 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-cnibin\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293722 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-lib-modules\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293732 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293762 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8ph\" (UniqueName: \"kubernetes.io/projected/958b5e80-9d2a-40f5-9975-f15dd89f4513-kube-api-access-tr8ph\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-node-log\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293816 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-var-lib-cni-multus\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293844 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-host\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293852 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293867 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15fee54d-02d3-4d29-a6de-d0c58007438d-ovnkube-script-lib\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.294268 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-socket-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293934 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvgcw\" (UniqueName: \"kubernetes.io/projected/95740bc6-ad16-43f7-8a66-1059c8800172-kube-api-access-hvgcw\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293965 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.293989 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-os-release\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294016 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-tuned\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-slash\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294080 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-var-lib-openvswitch\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294103 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15fee54d-02d3-4d29-a6de-d0c58007438d-env-overrides\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294157 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958b5e80-9d2a-40f5-9975-f15dd89f4513-tmp\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294202 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96afff64-0fa3-463f-b5f7-74d76396317d-iptables-alerter-script\") pod \"iptables-alerter-cs5dn\" (UID: \"96afff64-0fa3-463f-b5f7-74d76396317d\") " pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294229 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-systemd-units\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294334 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gbqtl\"" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294368 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp9m2\" (UniqueName: \"kubernetes.io/projected/96afff64-0fa3-463f-b5f7-74d76396317d-kube-api-access-vp9m2\") pod \"iptables-alerter-cs5dn\" (UID: \"96afff64-0fa3-463f-b5f7-74d76396317d\") " pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294403 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294437 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-run-k8s-cni-cncf-io\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-modprobe-d\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.295164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294485 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15fee54d-02d3-4d29-a6de-d0c58007438d-ovn-node-metrics-cert\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.296019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294509 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qxc\" (UniqueName: \"kubernetes.io/projected/15fee54d-02d3-4d29-a6de-d0c58007438d-kube-api-access-l9qxc\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.296019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294559 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96afff64-0fa3-463f-b5f7-74d76396317d-host-slash\") pod \"iptables-alerter-cs5dn\" (UID: \"96afff64-0fa3-463f-b5f7-74d76396317d\") " pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.296019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294594 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-registration-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.296019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-sys-fs\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.296019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-daemon-config\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.296019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294690 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-sysconfig\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.296019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-cni-bin\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.296019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294743 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-device-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.296019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.294782 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-var-lib-kubelet\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.318161 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.318131 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fx4rg" Jul 06 19:20:15.328394 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.326622 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fx4rg" Jul 06 19:20:15.381491 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.381415 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:20:15.395440 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7687a7fd-1cbf-4521-a462-e976d785f5e5-serviceca\") pod \"node-ca-tbczb\" (UID: \"7687a7fd-1cbf-4521-a462-e976d785f5e5\") " pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.395593 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395449 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvd4m\" (UniqueName: \"kubernetes.io/projected/726d29e8-475d-4f3c-90b1-cc52641bc44a-kube-api-access-kvd4m\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.395593 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-run-netns\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.395593 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.395593 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395566 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.395813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395597 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7687a7fd-1cbf-4521-a462-e976d785f5e5-host\") pod \"node-ca-tbczb\" (UID: \"7687a7fd-1cbf-4521-a462-e976d785f5e5\") " pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.395813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395621 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-system-cni-dir\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.395813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395648 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-hostroot\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.395813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-sysctl-d\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.395813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-run-systemd\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.395813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395724 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-etc-openvswitch\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.395813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395758 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15fee54d-02d3-4d29-a6de-d0c58007438d-ovnkube-config\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.395813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d99aa6b-369e-4f8e-984f-1b3097abfbcf-agent-certs\") pod \"konnectivity-agent-nfhbg\" (UID: \"5d99aa6b-369e-4f8e-984f-1b3097abfbcf\") " pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:15.395813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395805 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-run-netns\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395858 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-run-netns\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395859 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-hostroot\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395912 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-var-lib-cni-bin\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395936 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-run\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395956 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-sys\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395977 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7687a7fd-1cbf-4521-a462-e976d785f5e5-serviceca\") pod \"node-ca-tbczb\" (UID: \"7687a7fd-1cbf-4521-a462-e976d785f5e5\") " pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396010 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-sysctl-d\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.395979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d99aa6b-369e-4f8e-984f-1b3097abfbcf-konnectivity-ca\") pod \"konnectivity-agent-nfhbg\" (UID: \"5d99aa6b-369e-4f8e-984f-1b3097abfbcf\") " pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396043 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-systemd\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396060 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-run-systemd\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396060 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-run-netns\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396072 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-os-release\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396111 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-etc-openvswitch\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.396177 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:15.396247 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.396252 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret podName:44d4a5a8-5d12-483d-939c-5e40d5d3658b nodeName:}" failed. No retries permitted until 2026-07-06 19:20:15.896218208 +0000 UTC m=+2.956565118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret") pod "global-pull-secret-syncer-8rchd" (UID: "44d4a5a8-5d12-483d-939c-5e40d5d3658b") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396313 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396379 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7687a7fd-1cbf-4521-a462-e976d785f5e5-host\") pod \"node-ca-tbczb\" (UID: \"7687a7fd-1cbf-4521-a462-e976d785f5e5\") " pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396442 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-system-cni-dir\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396484 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-var-lib-cni-bin\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396528 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-run\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396574 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-sys\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396682 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15fee54d-02d3-4d29-a6de-d0c58007438d-ovnkube-config\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396797 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftwj\" (UniqueName: \"kubernetes.io/projected/7687a7fd-1cbf-4521-a462-e976d785f5e5-kube-api-access-zftwj\") pod \"node-ca-tbczb\" (UID: \"7687a7fd-1cbf-4521-a462-e976d785f5e5\") " pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396834 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-run-multus-certs\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396862 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-etc-kubernetes\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396889 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-kubernetes\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-sysctl-conf\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396943 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/726d29e8-475d-4f3c-90b1-cc52641bc44a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396973 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397001 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-cni-netd\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.396995 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397029 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-cnibin\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.397081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397055 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-lib-modules\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397069 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d99aa6b-369e-4f8e-984f-1b3097abfbcf-konnectivity-ca\") pod \"konnectivity-agent-nfhbg\" (UID: \"5d99aa6b-369e-4f8e-984f-1b3097abfbcf\") " pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397079 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8ph\" (UniqueName: \"kubernetes.io/projected/958b5e80-9d2a-40f5-9975-f15dd89f4513-kube-api-access-tr8ph\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397108 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2vt\" (UniqueName: \"kubernetes.io/projected/fe3d67f2-49f2-4ea8-8fab-29a1801d431a-kube-api-access-rx2vt\") pod \"node-resolver-cb5fx\" (UID: \"fe3d67f2-49f2-4ea8-8fab-29a1801d431a\") " pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397299 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-kubernetes\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-node-log\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397398 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-var-lib-cni-multus\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397424 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-host\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397454 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rcd6\" (UniqueName: \"kubernetes.io/projected/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-kube-api-access-2rcd6\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397481 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15fee54d-02d3-4d29-a6de-d0c58007438d-ovnkube-script-lib\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-socket-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397531 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvgcw\" (UniqueName: \"kubernetes.io/projected/95740bc6-ad16-43f7-8a66-1059c8800172-kube-api-access-hvgcw\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397559 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397572 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-sysctl-conf\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397584 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-os-release\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397626 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-tuned\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397636 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-cni-netd\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.397944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397643 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397662 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-slash\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-var-lib-openvswitch\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397684 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-cnibin\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397713 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15fee54d-02d3-4d29-a6de-d0c58007438d-env-overrides\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397721 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-var-lib-openvswitch\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.397805 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-os-release\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398120 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-node-log\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398168 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-systemd\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398233 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15fee54d-02d3-4d29-a6de-d0c58007438d-ovnkube-script-lib\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398289 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398329 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-var-lib-cni-multus\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398404 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-host\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398464 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-run-multus-certs\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398494 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-socket-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398528 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-etc-kubernetes\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.398793 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.398574 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958b5e80-9d2a-40f5-9975-f15dd89f4513-tmp\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.400868 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.400642 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-slash\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.401419 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.400782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96afff64-0fa3-463f-b5f7-74d76396317d-iptables-alerter-script\") pod \"iptables-alerter-cs5dn\" (UID: \"96afff64-0fa3-463f-b5f7-74d76396317d\") " pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.401419 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.401377 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d99aa6b-369e-4f8e-984f-1b3097abfbcf-agent-certs\") pod \"konnectivity-agent-nfhbg\" (UID: \"5d99aa6b-369e-4f8e-984f-1b3097abfbcf\") " pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:15.401705 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.401567 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-tuned\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.406929 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.406902 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftwj\" (UniqueName: \"kubernetes.io/projected/7687a7fd-1cbf-4521-a462-e976d785f5e5-kube-api-access-zftwj\") pod \"node-ca-tbczb\" (UID: \"7687a7fd-1cbf-4521-a462-e976d785f5e5\") " pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.407959 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.407650 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15fee54d-02d3-4d29-a6de-d0c58007438d-env-overrides\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.407959 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.407731 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-systemd-units\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.407959 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.407833 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-lib-modules\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.408165 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.407985 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-systemd-units\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.408165 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.407992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:15.408165 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408065 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9m2\" (UniqueName: \"kubernetes.io/projected/96afff64-0fa3-463f-b5f7-74d76396317d-kube-api-access-vp9m2\") pod \"iptables-alerter-cs5dn\" (UID: \"96afff64-0fa3-463f-b5f7-74d76396317d\") " pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.408165 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408115 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.408165 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408146 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-run-k8s-cni-cncf-io\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.408436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-modprobe-d\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.408436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408238 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15fee54d-02d3-4d29-a6de-d0c58007438d-ovn-node-metrics-cert\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.408436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408284 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qxc\" (UniqueName: \"kubernetes.io/projected/15fee54d-02d3-4d29-a6de-d0c58007438d-kube-api-access-l9qxc\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.408436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408312 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96afff64-0fa3-463f-b5f7-74d76396317d-host-slash\") pod \"iptables-alerter-cs5dn\" (UID: \"96afff64-0fa3-463f-b5f7-74d76396317d\") " pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.408436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408374 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-run-k8s-cni-cncf-io\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.408436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408414 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-registration-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.408436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408427 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-modprobe-d\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.408752 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408445 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96afff64-0fa3-463f-b5f7-74d76396317d-iptables-alerter-script\") pod \"iptables-alerter-cs5dn\" (UID: \"96afff64-0fa3-463f-b5f7-74d76396317d\") " pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.408752 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408462 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-sys-fs\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.408752 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408458 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96afff64-0fa3-463f-b5f7-74d76396317d-host-slash\") pod \"iptables-alerter-cs5dn\" (UID: \"96afff64-0fa3-463f-b5f7-74d76396317d\") " pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.408752 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408493 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-daemon-config\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.408752 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408414 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8ph\" (UniqueName: \"kubernetes.io/projected/958b5e80-9d2a-40f5-9975-f15dd89f4513-kube-api-access-tr8ph\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.408991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408783 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-sysconfig\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.408991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408840 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3d67f2-49f2-4ea8-8fab-29a1801d431a-tmp-dir\") pod \"node-resolver-cb5fx\" (UID: \"fe3d67f2-49f2-4ea8-8fab-29a1801d431a\") " pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.408991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408880 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-cni-bin\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.408991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408919 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-device-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.408991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408952 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-var-lib-kubelet\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.409293 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.408990 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-cnibin\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.409293 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409067 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/726d29e8-475d-4f3c-90b1-cc52641bc44a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.409293 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409120 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-kubelet\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.409293 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409152 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-var-lib-kubelet\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.409293 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409179 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/726d29e8-475d-4f3c-90b1-cc52641bc44a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.409293 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409219 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-run-ovn\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.409293 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-cni-binary-copy\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409309 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-conf-dir\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409375 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxbh\" (UniqueName: \"kubernetes.io/projected/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-kube-api-access-kpxbh\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409395 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-conf-dir\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409416 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe3d67f2-49f2-4ea8-8fab-29a1801d431a-hosts-file\") pod \"node-resolver-cb5fx\" (UID: \"fe3d67f2-49f2-4ea8-8fab-29a1801d431a\") " pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409453 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-run-openvswitch\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409488 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-run-ovn-kubernetes\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409537 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44d4a5a8-5d12-483d-939c-5e40d5d3658b-kubelet-config\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409555 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-run-ovn-kubernetes\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409575 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-cni-dir\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409611 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-socket-dir-parent\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409654 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-system-cni-dir\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409716 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-run-openvswitch\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.409740 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409726 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44d4a5a8-5d12-483d-939c-5e40d5d3658b-kubelet-config\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409811 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-cni-dir\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409871 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-log-socket\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409907 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44d4a5a8-5d12-483d-939c-5e40d5d3658b-dbus\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409928 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-daemon-config\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.409963 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-multus-socket-dir-parent\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410010 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-registration-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410044 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-host-var-lib-kubelet\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410054 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-log-socket\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410096 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-sys-fs\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410194 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44d4a5a8-5d12-483d-939c-5e40d5d3658b-dbus\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410098 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-var-lib-kubelet\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410221 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-run-ovn\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410312 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410363 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-cni-bin\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.410393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410369 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95740bc6-ad16-43f7-8a66-1059c8800172-device-dir\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.411030 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410418 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15fee54d-02d3-4d29-a6de-d0c58007438d-host-kubelet\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.411030 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410462 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/958b5e80-9d2a-40f5-9975-f15dd89f4513-etc-sysconfig\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.411030 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.410582 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-cni-binary-copy\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.413051 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.413014 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958b5e80-9d2a-40f5-9975-f15dd89f4513-tmp\") pod \"tuned-mkxfr\" (UID: \"958b5e80-9d2a-40f5-9975-f15dd89f4513\") " pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.413863 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.413839 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvgcw\" (UniqueName: \"kubernetes.io/projected/95740bc6-ad16-43f7-8a66-1059c8800172-kube-api-access-hvgcw\") pod \"aws-ebs-csi-driver-node-xnksh\" (UID: \"95740bc6-ad16-43f7-8a66-1059c8800172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.414659 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.414335 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:20:15.414659 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.414595 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:20:15.414659 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.414614 2571 projected.go:196] Error preparing data for projected volume kube-api-access-bnf6n for pod openshift-network-diagnostics/network-check-target-h4s4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:15.414897 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.414664 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15fee54d-02d3-4d29-a6de-d0c58007438d-ovn-node-metrics-cert\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.414897 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.414673 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n podName:8d5c45b9-a5f0-421d-bf44-59461dfd8fd0 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:15.914654927 +0000 UTC m=+2.975001854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bnf6n" (UniqueName: "kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n") pod "network-check-target-h4s4w" (UID: "8d5c45b9-a5f0-421d-bf44-59461dfd8fd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:15.416558 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.416536 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9m2\" (UniqueName: \"kubernetes.io/projected/96afff64-0fa3-463f-b5f7-74d76396317d-kube-api-access-vp9m2\") pod \"iptables-alerter-cs5dn\" (UID: \"96afff64-0fa3-463f-b5f7-74d76396317d\") " pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.417488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.417469 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxbh\" (UniqueName: \"kubernetes.io/projected/14c0f145-9474-4c4d-81b6-18fc6e2b11e8-kube-api-access-kpxbh\") pod \"multus-n57p8\" (UID: \"14c0f145-9474-4c4d-81b6-18fc6e2b11e8\") " pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.418389 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.418368 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qxc\" (UniqueName: \"kubernetes.io/projected/15fee54d-02d3-4d29-a6de-d0c58007438d-kube-api-access-l9qxc\") pod \"ovnkube-node-9xc2t\" (UID: \"15fee54d-02d3-4d29-a6de-d0c58007438d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.461830 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.461806 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:20:15.511054 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.510982 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3d67f2-49f2-4ea8-8fab-29a1801d431a-tmp-dir\") pod \"node-resolver-cb5fx\" (UID: \"fe3d67f2-49f2-4ea8-8fab-29a1801d431a\") " pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.511054 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511028 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-cnibin\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511232 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/726d29e8-475d-4f3c-90b1-cc52641bc44a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511232 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511100 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/726d29e8-475d-4f3c-90b1-cc52641bc44a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511232 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511121 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-cnibin\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511232 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511133 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe3d67f2-49f2-4ea8-8fab-29a1801d431a-hosts-file\") pod \"node-resolver-cb5fx\" (UID: \"fe3d67f2-49f2-4ea8-8fab-29a1801d431a\") " pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.511232 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511219 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe3d67f2-49f2-4ea8-8fab-29a1801d431a-hosts-file\") pod \"node-resolver-cb5fx\" (UID: \"fe3d67f2-49f2-4ea8-8fab-29a1801d431a\") " pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.511438 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511249 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-system-cni-dir\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511438 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511310 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvd4m\" (UniqueName: \"kubernetes.io/projected/726d29e8-475d-4f3c-90b1-cc52641bc44a-kube-api-access-kvd4m\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511438 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511319 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-system-cni-dir\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511438 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511366 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-os-release\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511438 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511388 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3d67f2-49f2-4ea8-8fab-29a1801d431a-tmp-dir\") pod \"node-resolver-cb5fx\" (UID: \"fe3d67f2-49f2-4ea8-8fab-29a1801d431a\") " pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.511438 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511398 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/726d29e8-475d-4f3c-90b1-cc52641bc44a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511438 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511424 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:15.511659 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2vt\" (UniqueName: \"kubernetes.io/projected/fe3d67f2-49f2-4ea8-8fab-29a1801d431a-kube-api-access-rx2vt\") pod \"node-resolver-cb5fx\" (UID: \"fe3d67f2-49f2-4ea8-8fab-29a1801d431a\") " pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.511659 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511471 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-os-release\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511659 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511482 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcd6\" (UniqueName: \"kubernetes.io/projected/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-kube-api-access-2rcd6\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:15.511659 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511512 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511790 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511655 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/726d29e8-475d-4f3c-90b1-cc52641bc44a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511790 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.511739 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/726d29e8-475d-4f3c-90b1-cc52641bc44a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.511790 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.511753 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:15.511909 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.511824 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs podName:3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:16.011805482 +0000 UTC m=+3.072152390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs") pod "network-metrics-daemon-gdvls" (UID: "3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:15.512136 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.512117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/726d29e8-475d-4f3c-90b1-cc52641bc44a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.512471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.512449 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/726d29e8-475d-4f3c-90b1-cc52641bc44a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.520633 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.520607 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2vt\" (UniqueName: \"kubernetes.io/projected/fe3d67f2-49f2-4ea8-8fab-29a1801d431a-kube-api-access-rx2vt\") pod \"node-resolver-cb5fx\" (UID: \"fe3d67f2-49f2-4ea8-8fab-29a1801d431a\") " pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.520633 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.520618 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvd4m\" (UniqueName: \"kubernetes.io/projected/726d29e8-475d-4f3c-90b1-cc52641bc44a-kube-api-access-kvd4m\") pod \"multus-additional-cni-plugins-hd6m4\" (UID: \"726d29e8-475d-4f3c-90b1-cc52641bc44a\") " pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.520763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.520738 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcd6\" (UniqueName: \"kubernetes.io/projected/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-kube-api-access-2rcd6\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:15.583802 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.583623 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" Jul 06 19:20:15.595735 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.595713 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:15.610524 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.610500 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cs5dn" Jul 06 19:20:15.616226 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.616206 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:15.624817 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.624800 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" Jul 06 19:20:15.633260 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.633243 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tbczb" Jul 06 19:20:15.641094 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.641072 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n57p8" Jul 06 19:20:15.650844 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.650820 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cb5fx" Jul 06 19:20:15.657429 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.657413 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" Jul 06 19:20:15.914470 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:15.914385 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:15.914598 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.914548 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:15.914642 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:15.914630 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret podName:44d4a5a8-5d12-483d-939c-5e40d5d3658b nodeName:}" failed. No retries permitted until 2026-07-06 19:20:16.914609272 +0000 UTC m=+3.974956199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret") pod "global-pull-secret-syncer-8rchd" (UID: "44d4a5a8-5d12-483d-939c-5e40d5d3658b") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:16.015008 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:16.014981 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:16.015160 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:16.015019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:16.015160 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:16.015123 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:20:16.015160 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:16.015123 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:16.015160 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:16.015136 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:20:16.015160 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:16.015146 2571 projected.go:196] Error preparing data for projected volume kube-api-access-bnf6n for pod openshift-network-diagnostics/network-check-target-h4s4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:16.015350 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:16.015182 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs podName:3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:17.015167548 +0000 UTC m=+4.075514466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs") pod "network-metrics-daemon-gdvls" (UID: "3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:16.015350 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:16.015195 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n podName:8d5c45b9-a5f0-421d-bf44-59461dfd8fd0 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:17.015189256 +0000 UTC m=+4.075536159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnf6n" (UniqueName: "kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n") pod "network-check-target-h4s4w" (UID: "8d5c45b9-a5f0-421d-bf44-59461dfd8fd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:16.329439 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:16.329361 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:15:15 +0000 UTC" deadline="2028-04-20 00:35:42.279738702 +0000 UTC" Jul 06 19:20:16.329439 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:16.329388 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15677h15m25.950354208s" Jul 06 19:20:16.455177 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:16.455145 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:16.455326 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:16.455262 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:16.558622 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:16.558599 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7687a7fd_1cbf_4521_a462_e976d785f5e5.slice/crio-d89d51c7f7eb7d328079ae96fb85c3def10e5f4695859da30cad2019e95e17f9 WatchSource:0}: Error finding container d89d51c7f7eb7d328079ae96fb85c3def10e5f4695859da30cad2019e95e17f9: Status 404 returned error can't find the container with id d89d51c7f7eb7d328079ae96fb85c3def10e5f4695859da30cad2019e95e17f9 Jul 06 19:20:16.560466 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:16.560446 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3d67f2_49f2_4ea8_8fab_29a1801d431a.slice/crio-762d330b3a40711afdf7eb56d84d106ef1cc9275ac34a62ffd9bdf0ccab6d049 WatchSource:0}: Error finding container 762d330b3a40711afdf7eb56d84d106ef1cc9275ac34a62ffd9bdf0ccab6d049: Status 404 returned error can't find the container with id 762d330b3a40711afdf7eb56d84d106ef1cc9275ac34a62ffd9bdf0ccab6d049 Jul 06 19:20:16.562331 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:16.562301 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d99aa6b_369e_4f8e_984f_1b3097abfbcf.slice/crio-1b7accf2297b963d4c057d8f62bad4fc7a816002b6e7dc154bbddc404e0bc657 WatchSource:0}: Error finding container 1b7accf2297b963d4c057d8f62bad4fc7a816002b6e7dc154bbddc404e0bc657: Status 404 returned error can't find the container with id 1b7accf2297b963d4c057d8f62bad4fc7a816002b6e7dc154bbddc404e0bc657 Jul 06 19:20:16.565337 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:16.565314 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726d29e8_475d_4f3c_90b1_cc52641bc44a.slice/crio-50ac0148908a7fb7f0bf4e99857050f66ca6229076fd6202451ad19064d96308 WatchSource:0}: Error finding container 50ac0148908a7fb7f0bf4e99857050f66ca6229076fd6202451ad19064d96308: Status 404 returned error can't find the container with id 50ac0148908a7fb7f0bf4e99857050f66ca6229076fd6202451ad19064d96308 Jul 06 19:20:16.566569 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:16.566546 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fee54d_02d3_4d29_a6de_d0c58007438d.slice/crio-5ad1f85ef91b850f5b8c212b923ace3f8ec177c9505d018f1da1638763630631 WatchSource:0}: Error finding container 5ad1f85ef91b850f5b8c212b923ace3f8ec177c9505d018f1da1638763630631: Status 404 returned error can't find the container with id 5ad1f85ef91b850f5b8c212b923ace3f8ec177c9505d018f1da1638763630631 Jul 06 19:20:16.568741 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:16.568537 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96afff64_0fa3_463f_b5f7_74d76396317d.slice/crio-255be18f4916b92db3dde9959799e879fa95a5c1dcb60c00891b0a1342bb0e40 WatchSource:0}: Error finding container 255be18f4916b92db3dde9959799e879fa95a5c1dcb60c00891b0a1342bb0e40: Status 404 returned error can't find the container with id 255be18f4916b92db3dde9959799e879fa95a5c1dcb60c00891b0a1342bb0e40 Jul 06 19:20:16.568961 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:16.568880 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95740bc6_ad16_43f7_8a66_1059c8800172.slice/crio-20b2757487a557682cb21dfc5d0750b6614d52b921ef702b68366ba83f7f8eaf WatchSource:0}: Error finding container 20b2757487a557682cb21dfc5d0750b6614d52b921ef702b68366ba83f7f8eaf: Status 404 returned error can't find the container with id 20b2757487a557682cb21dfc5d0750b6614d52b921ef702b68366ba83f7f8eaf Jul 06 19:20:16.570025 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:16.569922 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958b5e80_9d2a_40f5_9975_f15dd89f4513.slice/crio-2918b4050b4f80d88692bf444da1a1caa3fdc1270c80ab534910068e1d0528f9 WatchSource:0}: Error finding container 2918b4050b4f80d88692bf444da1a1caa3fdc1270c80ab534910068e1d0528f9: Status 404 returned error can't find the container with id 2918b4050b4f80d88692bf444da1a1caa3fdc1270c80ab534910068e1d0528f9 Jul 06 19:20:16.573174 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:20:16.573154 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c0f145_9474_4c4d_81b6_18fc6e2b11e8.slice/crio-80a491154fad47f98d1b80aa739916ee4db3c85f6904a9f73e4acded055c50fa WatchSource:0}: Error finding container 80a491154fad47f98d1b80aa739916ee4db3c85f6904a9f73e4acded055c50fa: Status 404 returned error can't find the container with id 80a491154fad47f98d1b80aa739916ee4db3c85f6904a9f73e4acded055c50fa Jul 06 19:20:16.921238 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:16.921211 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:16.921381 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:16.921359 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:16.921505 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:16.921414 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret podName:44d4a5a8-5d12-483d-939c-5e40d5d3658b nodeName:}" failed. No retries permitted until 2026-07-06 19:20:18.921395687 +0000 UTC m=+5.981742600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret") pod "global-pull-secret-syncer-8rchd" (UID: "44d4a5a8-5d12-483d-939c-5e40d5d3658b") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:17.022368 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.022340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:17.022548 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.022411 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:17.022548 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:17.022505 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:17.022548 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:17.022533 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:20:17.022652 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:17.022556 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:20:17.022652 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:17.022570 2571 projected.go:196] Error preparing data for projected volume kube-api-access-bnf6n for pod openshift-network-diagnostics/network-check-target-h4s4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:17.022652 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:17.022558 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs podName:3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:19.022539723 +0000 UTC m=+6.082886646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs") pod "network-metrics-daemon-gdvls" (UID: "3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:17.022652 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:17.022631 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n podName:8d5c45b9-a5f0-421d-bf44-59461dfd8fd0 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:19.022614623 +0000 UTC m=+6.082961532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnf6n" (UniqueName: "kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n") pod "network-check-target-h4s4w" (UID: "8d5c45b9-a5f0-421d-bf44-59461dfd8fd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:17.331604 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.329676 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:15:15 +0000 UTC" deadline="2028-03-31 00:13:15.604829258 +0000 UTC" Jul 06 19:20:17.331604 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.329734 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15196h52m58.275099593s" Jul 06 19:20:17.456448 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.456403 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:17.456634 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:17.456525 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:17.456888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.456414 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:17.456888 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:17.456843 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:17.487292 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.484721 2571 generic.go:356] "Generic (PLEG): container finished" podID="ee9ea7bd3edf3de498f018682e7d0a02" containerID="b58a7e9ec27a73bd3402437a9a99917b506d1335b6ffe4a17ac181af770efb2b" exitCode=0 Jul 06 19:20:17.487292 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.484886 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" event={"ID":"ee9ea7bd3edf3de498f018682e7d0a02","Type":"ContainerDied","Data":"b58a7e9ec27a73bd3402437a9a99917b506d1335b6ffe4a17ac181af770efb2b"} Jul 06 19:20:17.502763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.502733 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" event={"ID":"958b5e80-9d2a-40f5-9975-f15dd89f4513","Type":"ContainerStarted","Data":"2918b4050b4f80d88692bf444da1a1caa3fdc1270c80ab534910068e1d0528f9"} Jul 06 19:20:17.515463 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.515436 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" event={"ID":"95740bc6-ad16-43f7-8a66-1059c8800172","Type":"ContainerStarted","Data":"20b2757487a557682cb21dfc5d0750b6614d52b921ef702b68366ba83f7f8eaf"} Jul 06 19:20:17.521690 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.521665 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" event={"ID":"726d29e8-475d-4f3c-90b1-cc52641bc44a","Type":"ContainerStarted","Data":"50ac0148908a7fb7f0bf4e99857050f66ca6229076fd6202451ad19064d96308"} Jul 06 19:20:17.545683 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.545648 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cb5fx" event={"ID":"fe3d67f2-49f2-4ea8-8fab-29a1801d431a","Type":"ContainerStarted","Data":"762d330b3a40711afdf7eb56d84d106ef1cc9275ac34a62ffd9bdf0ccab6d049"} Jul 06 19:20:17.561049 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.560853 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tbczb" event={"ID":"7687a7fd-1cbf-4521-a462-e976d785f5e5","Type":"ContainerStarted","Data":"d89d51c7f7eb7d328079ae96fb85c3def10e5f4695859da30cad2019e95e17f9"} Jul 06 19:20:17.580586 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.580556 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal" event={"ID":"cc1d303b9ebb2e2da2b7750642266d77","Type":"ContainerStarted","Data":"1471cadb63b812b96568e8d570da0bcd3c820ae9aefa298166b3080723f9422c"} Jul 06 19:20:17.599020 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.598952 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n57p8" event={"ID":"14c0f145-9474-4c4d-81b6-18fc6e2b11e8","Type":"ContainerStarted","Data":"80a491154fad47f98d1b80aa739916ee4db3c85f6904a9f73e4acded055c50fa"} Jul 06 19:20:17.611078 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.611051 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cs5dn" event={"ID":"96afff64-0fa3-463f-b5f7-74d76396317d","Type":"ContainerStarted","Data":"255be18f4916b92db3dde9959799e879fa95a5c1dcb60c00891b0a1342bb0e40"} Jul 06 19:20:17.631633 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.631609 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" event={"ID":"15fee54d-02d3-4d29-a6de-d0c58007438d","Type":"ContainerStarted","Data":"5ad1f85ef91b850f5b8c212b923ace3f8ec177c9505d018f1da1638763630631"} Jul 06 19:20:17.655037 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:17.654999 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nfhbg" event={"ID":"5d99aa6b-369e-4f8e-984f-1b3097abfbcf","Type":"ContainerStarted","Data":"1b7accf2297b963d4c057d8f62bad4fc7a816002b6e7dc154bbddc404e0bc657"} Jul 06 19:20:18.456192 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:18.455386 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:18.456192 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:18.455508 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:18.680520 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:18.680483 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" event={"ID":"ee9ea7bd3edf3de498f018682e7d0a02","Type":"ContainerStarted","Data":"0356faa672ac509eb0b3e96c3485b5217072765b086914b09c89797046116978"} Jul 06 19:20:18.696639 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:18.696586 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-211.ec2.internal" podStartSLOduration=4.696566583 podStartE2EDuration="4.696566583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:18.696328395 +0000 UTC m=+5.756675322" watchObservedRunningTime="2026-07-06 19:20:18.696566583 +0000 UTC m=+5.756913529" Jul 06 19:20:18.696910 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:18.696868 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-211.ec2.internal" podStartSLOduration=4.69685803 podStartE2EDuration="4.69685803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:17.596796712 +0000 UTC m=+4.657143644" watchObservedRunningTime="2026-07-06 19:20:18.69685803 +0000 UTC m=+5.757204958" Jul 06 19:20:18.939586 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:18.939506 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:18.939739 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:18.939676 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:18.939739 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:18.939736 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret podName:44d4a5a8-5d12-483d-939c-5e40d5d3658b nodeName:}" failed. No retries permitted until 2026-07-06 19:20:22.939719273 +0000 UTC m=+10.000066180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret") pod "global-pull-secret-syncer-8rchd" (UID: "44d4a5a8-5d12-483d-939c-5e40d5d3658b") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:19.040392 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:19.040345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:19.040547 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:19.040409 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:19.040621 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:19.040548 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:20:19.040621 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:19.040562 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:20:19.040621 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:19.040593 2571 projected.go:196] Error preparing data for projected volume kube-api-access-bnf6n for pod openshift-network-diagnostics/network-check-target-h4s4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:19.040762 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:19.040683 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n podName:8d5c45b9-a5f0-421d-bf44-59461dfd8fd0 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:23.040664206 +0000 UTC m=+10.101011130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnf6n" (UniqueName: "kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n") pod "network-check-target-h4s4w" (UID: "8d5c45b9-a5f0-421d-bf44-59461dfd8fd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:19.041125 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:19.041104 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:19.041225 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:19.041159 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs podName:3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:23.041144362 +0000 UTC m=+10.101491280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs") pod "network-metrics-daemon-gdvls" (UID: "3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:19.455388 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:19.455362 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:19.455559 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:19.455488 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:19.455826 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:19.455808 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:19.455933 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:19.455912 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:20.455491 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:20.455456 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:20.455887 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:20.455603 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:21.456405 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:21.455711 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:21.456405 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:21.455850 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:21.456405 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:21.456256 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:21.456405 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:21.456360 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:22.455421 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:22.455395 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:22.455564 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:22.455492 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:22.969543 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:22.969505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:22.969968 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:22.969686 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:22.969968 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:22.969754 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret podName:44d4a5a8-5d12-483d-939c-5e40d5d3658b nodeName:}" failed. No retries permitted until 2026-07-06 19:20:30.969735018 +0000 UTC m=+18.030081939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret") pod "global-pull-secret-syncer-8rchd" (UID: "44d4a5a8-5d12-483d-939c-5e40d5d3658b") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:23.070127 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:23.070089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:23.070309 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:23.070147 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:23.070384 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:23.070349 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:20:23.070384 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:23.070369 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:20:23.070384 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:23.070381 2571 projected.go:196] Error preparing data for projected volume kube-api-access-bnf6n for pod openshift-network-diagnostics/network-check-target-h4s4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:23.070504 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:23.070424 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n podName:8d5c45b9-a5f0-421d-bf44-59461dfd8fd0 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:31.070410698 +0000 UTC m=+18.130757615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnf6n" (UniqueName: "kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n") pod "network-check-target-h4s4w" (UID: "8d5c45b9-a5f0-421d-bf44-59461dfd8fd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:23.070872 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:23.070857 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:23.070917 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:23.070907 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs podName:3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:31.070892089 +0000 UTC m=+18.131238998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs") pod "network-metrics-daemon-gdvls" (UID: "3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:23.456159 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:23.456078 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:23.456325 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:23.456181 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:23.456583 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:23.456566 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:23.456725 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:23.456673 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:24.455702 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:24.455623 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:24.456137 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:24.455750 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:25.455906 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:25.455875 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:25.456372 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:25.455882 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:25.456372 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:25.455990 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:25.456372 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:25.456069 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:26.454862 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.454838 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:26.455045 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:26.455024 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:26.700336 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.700079 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" event={"ID":"726d29e8-475d-4f3c-90b1-cc52641bc44a","Type":"ContainerStarted","Data":"ed98e9ff3e16467f784dea4bc17ca473e3f0ec819ed2610f2a75126f8e96a5f2"} Jul 06 19:20:26.704175 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.703836 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cb5fx" event={"ID":"fe3d67f2-49f2-4ea8-8fab-29a1801d431a","Type":"ContainerStarted","Data":"db6c487c84cdb1cf9e0dfee15939fc6262e59f86b33411e7df4ee284ae1def90"} Jul 06 19:20:26.706318 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.706239 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tbczb" event={"ID":"7687a7fd-1cbf-4521-a462-e976d785f5e5","Type":"ContainerStarted","Data":"a1e78316ed23ef6ad7821d7eb2924983654716aaa502a993fdd64850d2848074"} Jul 06 19:20:26.708143 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.707810 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nfhbg" event={"ID":"5d99aa6b-369e-4f8e-984f-1b3097abfbcf","Type":"ContainerStarted","Data":"8a46fd405f261e41c226eeeb90c619281517bd03394c46917d2aec15fa77156f"} Jul 06 19:20:26.710869 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.709884 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" event={"ID":"958b5e80-9d2a-40f5-9975-f15dd89f4513","Type":"ContainerStarted","Data":"811344679ead860ac0282ad63aeba75620f9d83f6f8d89c6364b346b3d24f2d9"} Jul 06 19:20:26.712310 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.712218 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" event={"ID":"95740bc6-ad16-43f7-8a66-1059c8800172","Type":"ContainerStarted","Data":"ea003b7b19af369bbf28f175bd0eb7c4f581546664e75f8c3461c1f52aa90a42"} Jul 06 19:20:26.738008 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.737965 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mkxfr" podStartSLOduration=3.837310064 podStartE2EDuration="13.737950514s" totalImagesPullingTime="9.90064045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:26.737490959 +0000 UTC m=+13.797837885" watchObservedRunningTime="2026-07-06 19:20:26.737950514 +0000 UTC m=+13.798297440" Jul 06 19:20:26.752634 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.752591 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nfhbg" podStartSLOduration=3.898810702 podStartE2EDuration="13.752573693s" totalImagesPullingTime="9.853762991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:26.752202698 +0000 UTC m=+13.812549624" watchObservedRunningTime="2026-07-06 19:20:26.752573693 +0000 UTC m=+13.812920620" Jul 06 19:20:26.766897 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.766844 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cb5fx" podStartSLOduration=3.906149085 podStartE2EDuration="13.766829947s" totalImagesPullingTime="9.860680862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:26.766658112 +0000 UTC m=+13.827005039" watchObservedRunningTime="2026-07-06 19:20:26.766829947 +0000 UTC m=+13.827176874" Jul 06 19:20:26.782926 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:26.782880 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tbczb" podStartSLOduration=3.896153473 podStartE2EDuration="13.782862674s" totalImagesPullingTime="9.886709201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:26.782798026 +0000 UTC m=+13.843144953" watchObservedRunningTime="2026-07-06 19:20:26.782862674 +0000 UTC m=+13.843209603" Jul 06 19:20:27.455519 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:27.455489 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:27.455519 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:27.455523 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:27.455733 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:27.455613 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:27.455796 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:27.455725 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:27.719195 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:27.716773 2571 generic.go:356] "Generic (PLEG): container finished" podID="726d29e8-475d-4f3c-90b1-cc52641bc44a" containerID="ed98e9ff3e16467f784dea4bc17ca473e3f0ec819ed2610f2a75126f8e96a5f2" exitCode=0 Jul 06 19:20:27.719195 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:27.717874 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" event={"ID":"726d29e8-475d-4f3c-90b1-cc52641bc44a","Type":"ContainerDied","Data":"ed98e9ff3e16467f784dea4bc17ca473e3f0ec819ed2610f2a75126f8e96a5f2"} Jul 06 19:20:27.981261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:27.981103 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:20:28.347674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:28.347513 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:20:27.981259133Z","UUID":"368edfb3-0439-4e48-9cd6-0640c7ea9207","Handler":null,"Name":"","Endpoint":""} Jul 06 19:20:28.349596 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:28.349571 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:20:28.349721 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:28.349645 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:20:28.455056 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:28.455024 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:28.455215 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:28.455129 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:28.721050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:28.720966 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cs5dn" event={"ID":"96afff64-0fa3-463f-b5f7-74d76396317d","Type":"ContainerStarted","Data":"36e767e51ed1dd63529b5b81ea51ad9d78706fc32467ae6f3117aaf638444d0b"} Jul 06 19:20:28.723258 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:28.723231 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" event={"ID":"95740bc6-ad16-43f7-8a66-1059c8800172","Type":"ContainerStarted","Data":"7aebcea9ce36f416104f3ffc0a778544dc52a8047b72e4ee73db6c743fcc099b"} Jul 06 19:20:29.455406 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:29.455374 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:29.455585 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:29.455488 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:29.455874 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:29.455856 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:29.455975 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:29.455953 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:29.686401 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:29.686370 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:29.687739 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:29.687579 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:29.703481 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:29.703436 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cs5dn" podStartSLOduration=6.823321365 podStartE2EDuration="16.703395548s" totalImagesPullingTime="9.880074183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:28.736818244 +0000 UTC m=+15.797165172" watchObservedRunningTime="2026-07-06 19:20:29.703395548 +0000 UTC m=+16.763742475" Jul 06 19:20:29.725518 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:29.725437 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:29.725887 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:29.725755 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nfhbg" Jul 06 19:20:30.455073 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:30.455042 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:30.455216 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:30.455156 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:31.030291 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:31.030237 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:31.030704 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.030401 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:31.030704 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.030479 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret podName:44d4a5a8-5d12-483d-939c-5e40d5d3658b nodeName:}" failed. No retries permitted until 2026-07-06 19:20:47.030463442 +0000 UTC m=+34.090810350 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret") pod "global-pull-secret-syncer-8rchd" (UID: "44d4a5a8-5d12-483d-939c-5e40d5d3658b") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:31.130920 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:31.130873 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:31.131075 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:31.130933 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:31.131075 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.131045 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:31.131208 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.131091 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:20:31.131208 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.131105 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs podName:3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:47.131089801 +0000 UTC m=+34.191436710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs") pod "network-metrics-daemon-gdvls" (UID: "3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:31.131208 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.131107 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:20:31.131208 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.131122 2571 projected.go:196] Error preparing data for projected volume kube-api-access-bnf6n for pod openshift-network-diagnostics/network-check-target-h4s4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:31.131208 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.131171 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n podName:8d5c45b9-a5f0-421d-bf44-59461dfd8fd0 nodeName:}" failed. No retries permitted until 2026-07-06 19:20:47.131156263 +0000 UTC m=+34.191503182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnf6n" (UniqueName: "kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n") pod "network-check-target-h4s4w" (UID: "8d5c45b9-a5f0-421d-bf44-59461dfd8fd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:31.456555 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:31.456478 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:31.456695 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.456617 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:31.456695 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:31.456642 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:31.456782 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:31.456752 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:32.455911 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:32.455880 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:32.456402 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:32.456005 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:33.455790 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:33.455751 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:33.455938 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:33.455870 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:33.455938 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:33.455924 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:33.456325 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:33.456046 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:34.455029 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:34.455001 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:34.455180 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:34.455126 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:35.455813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:35.455778 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:35.456380 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:35.455825 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:35.456380 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:35.455908 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:35.456380 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:35.456070 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:35.561521 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:35.561494 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cb5fx_fe3d67f2-49f2-4ea8-8fab-29a1801d431a/dns-node-resolver/0.log" Jul 06 19:20:36.455641 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:36.455610 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:36.455824 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:36.455745 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:36.729872 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:36.729798 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tbczb_7687a7fd-1cbf-4521-a462-e976d785f5e5/node-ca/0.log" Jul 06 19:20:37.455122 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:37.455093 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:37.455300 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:37.455223 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:37.455300 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:37.455290 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:37.455430 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:37.455406 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:38.455344 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:38.455314 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:38.455762 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:38.455409 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:39.455125 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.454929 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:39.455226 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.454993 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:39.455317 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:39.455242 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:39.455397 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:39.455377 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:39.747476 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.747396 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" event={"ID":"95740bc6-ad16-43f7-8a66-1059c8800172","Type":"ContainerStarted","Data":"81a1a00019fa5177d3c5dfa43502738121e5f3edbe1b8e629e8fcccb41ea569f"} Jul 06 19:20:39.749017 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.748993 2571 generic.go:356] "Generic (PLEG): container finished" podID="726d29e8-475d-4f3c-90b1-cc52641bc44a" containerID="65c3fedccead944a88c8ede7ef949a9d5d8e5d2cee8a7eb85576faf85263077c" exitCode=0 Jul 06 19:20:39.749131 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.749057 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" event={"ID":"726d29e8-475d-4f3c-90b1-cc52641bc44a","Type":"ContainerDied","Data":"65c3fedccead944a88c8ede7ef949a9d5d8e5d2cee8a7eb85576faf85263077c"} Jul 06 19:20:39.750559 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.750531 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n57p8" event={"ID":"14c0f145-9474-4c4d-81b6-18fc6e2b11e8","Type":"ContainerStarted","Data":"a4d9f0b2cb62ccc264513fa2e9c27e017fad36f22b7c85902a65752eb00031fe"} Jul 06 19:20:39.755620 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.755601 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" event={"ID":"15fee54d-02d3-4d29-a6de-d0c58007438d","Type":"ContainerStarted","Data":"d88ad52480c5186d399c4140c786149745bbd28f0d9542797082176ea94a5c80"} Jul 06 19:20:39.755699 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.755654 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" event={"ID":"15fee54d-02d3-4d29-a6de-d0c58007438d","Type":"ContainerStarted","Data":"f738121ef4152ba2fd66d6113b37c292ebc3d026872715ecd88b304c9996971e"} Jul 06 19:20:39.755699 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.755683 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" event={"ID":"15fee54d-02d3-4d29-a6de-d0c58007438d","Type":"ContainerStarted","Data":"9897b23c159276531d6bdfd206a25cd194408bac75d9c73ac729cc3487b0c446"} Jul 06 19:20:39.755769 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.755710 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" event={"ID":"15fee54d-02d3-4d29-a6de-d0c58007438d","Type":"ContainerStarted","Data":"e84fc521256cc83d3a362ad2767d435cf7eb1df4d18bde7be1695af3c28c0f9b"} Jul 06 19:20:39.755769 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.755738 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" event={"ID":"15fee54d-02d3-4d29-a6de-d0c58007438d","Type":"ContainerStarted","Data":"74b708ed7838ed577780de40642f3ba6fe8e2fa12e15c38d1bca952ce5ba0f9b"} Jul 06 19:20:39.755769 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.755765 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" event={"ID":"15fee54d-02d3-4d29-a6de-d0c58007438d","Type":"ContainerStarted","Data":"c10f65a51c4c4188ab28530cf5f7eff2f4ff1abfee221e7775ab1720cc0abaf6"} Jul 06 19:20:39.771926 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.771887 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnksh" podStartSLOduration=4.624551565 podStartE2EDuration="26.771874993s" totalImagesPullingTime="22.147323428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:39.770598519 +0000 UTC m=+26.830945444" watchObservedRunningTime="2026-07-06 19:20:39.771874993 +0000 UTC m=+26.832221919" Jul 06 19:20:39.826184 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:39.826142 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n57p8" podStartSLOduration=4.516228075 podStartE2EDuration="26.826127544s" totalImagesPullingTime="22.309899469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:39.794164035 +0000 UTC m=+26.854510961" watchObservedRunningTime="2026-07-06 19:20:39.826127544 +0000 UTC m=+26.886474470" Jul 06 19:20:40.455824 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:40.455802 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:40.456149 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:40.455895 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:40.760203 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:40.760122 2571 generic.go:356] "Generic (PLEG): container finished" podID="726d29e8-475d-4f3c-90b1-cc52641bc44a" containerID="52e16372f2241508bae2e63557fd466dc9fec5ec2c2e28a323e9e64a4dca0e8e" exitCode=0 Jul 06 19:20:40.760351 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:40.760207 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" event={"ID":"726d29e8-475d-4f3c-90b1-cc52641bc44a","Type":"ContainerDied","Data":"52e16372f2241508bae2e63557fd466dc9fec5ec2c2e28a323e9e64a4dca0e8e"} Jul 06 19:20:41.455233 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:41.455013 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:41.455403 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:41.455070 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:41.455403 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:41.455304 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:41.455506 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:41.455399 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:41.766399 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:41.766372 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" event={"ID":"15fee54d-02d3-4d29-a6de-d0c58007438d","Type":"ContainerStarted","Data":"b57d72eb6b86b03f51b0fdf4d10bd8f690a9ce6314556ab3e2988aa9115ecec9"} Jul 06 19:20:41.769033 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:41.769008 2571 generic.go:356] "Generic (PLEG): container finished" podID="726d29e8-475d-4f3c-90b1-cc52641bc44a" containerID="93b83887d3311c4d8e17efe6c58c125434c21aa723b5b5c7c0f66d234de1fa01" exitCode=0 Jul 06 19:20:41.769136 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:41.769050 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" event={"ID":"726d29e8-475d-4f3c-90b1-cc52641bc44a","Type":"ContainerDied","Data":"93b83887d3311c4d8e17efe6c58c125434c21aa723b5b5c7c0f66d234de1fa01"} Jul 06 19:20:42.455822 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:42.455797 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:42.455992 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:42.455880 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:43.455869 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:43.455832 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:43.456251 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:43.455959 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:43.456251 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:43.456073 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:43.456251 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:43.456186 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:44.455205 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:44.455171 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:44.455491 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:44.455328 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:44.778254 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:44.778179 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" event={"ID":"15fee54d-02d3-4d29-a6de-d0c58007438d","Type":"ContainerStarted","Data":"a1413b905298d0eb21e8e173cf867dc4c00a383ef5b33b9590471af0b9b2ae8d"} Jul 06 19:20:44.778697 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:44.778374 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:44.778697 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:44.778459 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:44.778697 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:44.778526 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:44.794672 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:44.794648 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:44.794817 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:44.794779 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:20:44.806019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:44.805971 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" podStartSLOduration=9.307170161 podStartE2EDuration="31.805954248s" totalImagesPullingTime="22.498784087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:44.804995757 +0000 UTC m=+31.865342683" watchObservedRunningTime="2026-07-06 19:20:44.805954248 +0000 UTC m=+31.866301175" Jul 06 19:20:45.455073 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:45.455042 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:45.455252 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:45.455145 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:45.455252 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:45.455205 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:45.455399 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:45.455344 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:46.454918 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:46.454882 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:46.455343 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:46.454990 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:47.047752 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:47.047722 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:47.047905 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.047825 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:47.047905 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.047871 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret podName:44d4a5a8-5d12-483d-939c-5e40d5d3658b nodeName:}" failed. No retries permitted until 2026-07-06 19:21:19.04785772 +0000 UTC m=+66.108204624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret") pod "global-pull-secret-syncer-8rchd" (UID: "44d4a5a8-5d12-483d-939c-5e40d5d3658b") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:20:47.148775 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:47.148744 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:47.148905 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:47.148796 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:47.148905 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.148869 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:47.148905 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.148876 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:20:47.148905 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.148895 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:20:47.148905 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.148904 2571 projected.go:196] Error preparing data for projected volume kube-api-access-bnf6n for pod openshift-network-diagnostics/network-check-target-h4s4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:47.149066 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.148923 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs podName:3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6 nodeName:}" failed. No retries permitted until 2026-07-06 19:21:19.148901342 +0000 UTC m=+66.209248245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs") pod "network-metrics-daemon-gdvls" (UID: "3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:20:47.149066 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.148949 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n podName:8d5c45b9-a5f0-421d-bf44-59461dfd8fd0 nodeName:}" failed. No retries permitted until 2026-07-06 19:21:19.148937042 +0000 UTC m=+66.209283946 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnf6n" (UniqueName: "kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n") pod "network-check-target-h4s4w" (UID: "8d5c45b9-a5f0-421d-bf44-59461dfd8fd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:20:47.455285 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:47.455106 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:47.455615 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:47.455162 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:47.455615 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.455370 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:47.455615 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:47.455486 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:47.788329 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:47.788237 2571 generic.go:356] "Generic (PLEG): container finished" podID="726d29e8-475d-4f3c-90b1-cc52641bc44a" containerID="4749e81d235243871cff8f32b40dd92831551da215366f236ccf9c6ee03b80cc" exitCode=0 Jul 06 19:20:47.788329 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:47.788292 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" event={"ID":"726d29e8-475d-4f3c-90b1-cc52641bc44a","Type":"ContainerDied","Data":"4749e81d235243871cff8f32b40dd92831551da215366f236ccf9c6ee03b80cc"} Jul 06 19:20:48.455479 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:48.455452 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:48.455840 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:48.455551 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:48.793617 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:48.793539 2571 generic.go:356] "Generic (PLEG): container finished" podID="726d29e8-475d-4f3c-90b1-cc52641bc44a" containerID="af4f6a3cb6c3e17836b828e1fcea6af0aad60c522ff3b7c9b8dbfefa668dbd4f" exitCode=0 Jul 06 19:20:48.793617 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:48.793574 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" event={"ID":"726d29e8-475d-4f3c-90b1-cc52641bc44a","Type":"ContainerDied","Data":"af4f6a3cb6c3e17836b828e1fcea6af0aad60c522ff3b7c9b8dbfefa668dbd4f"} Jul 06 19:20:49.457938 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:49.457912 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:49.458378 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:49.458003 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:49.458378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:49.458058 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:49.458378 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:49.458126 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:49.799149 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:49.799067 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" event={"ID":"726d29e8-475d-4f3c-90b1-cc52641bc44a","Type":"ContainerStarted","Data":"d40a2847d0846bfa17a546b5a9a1ed6b1358ad627c419f63b166601e8e2f01f5"} Jul 06 19:20:49.823662 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:49.823620 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hd6m4" podStartSLOduration=8.944728219 podStartE2EDuration="36.823605763s" totalImagesPullingTime="27.878877544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:20:49.822611351 +0000 UTC m=+36.882958277" watchObservedRunningTime="2026-07-06 19:20:49.823605763 +0000 UTC m=+36.883952712" Jul 06 19:20:50.455853 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:50.455823 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:50.456025 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:50.455919 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:51.455329 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:51.455295 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:51.455777 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:51.455336 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:51.455777 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:51.455427 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:51.455777 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:51.455529 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:52.455332 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:52.455302 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:52.455776 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:52.455413 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:53.456369 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:53.456340 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:53.456749 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:53.456419 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:53.456749 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:53.456425 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:53.456749 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:53.456499 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:54.455176 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:54.455147 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:54.455363 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:54.455236 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:55.455482 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:55.455448 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:55.455979 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:55.455454 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:55.455979 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:55.455561 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:55.455979 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:55.455665 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:56.455854 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:56.455826 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:56.456215 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:56.455921 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:57.455229 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:57.455202 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:57.455229 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:57.455215 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:57.455493 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:57.455339 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:57.455493 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:57.455409 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:58.454992 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:58.454958 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:58.455380 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:58.455055 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:20:59.455022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:59.454989 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:59.455449 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:59.454992 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:59.455449 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:59.455139 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:59.455449 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:59.455211 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:59.975969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:59.975928 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gdvls"] Jul 06 19:20:59.976119 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:59.976061 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:20:59.976200 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:59.976173 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:20:59.976652 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:59.976628 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4s4w"] Jul 06 19:20:59.976778 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:59.976719 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:20:59.976840 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:59.976826 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:20:59.977261 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:59.977242 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8rchd"] Jul 06 19:20:59.977351 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:20:59.977333 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:20:59.977425 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:20:59.977409 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:21:01.455100 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:01.455072 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:21:01.455709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:01.455072 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:21:01.455709 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:01.455186 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:21:01.455709 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:01.455299 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:21:02.455737 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:02.455572 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:21:02.456033 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:02.455813 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:21:03.456294 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:03.456239 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:21:03.456731 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:03.456339 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:21:03.456731 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:03.456353 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdvls" podUID="3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6" Jul 06 19:21:03.456731 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:03.456454 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rchd" podUID="44d4a5a8-5d12-483d-939c-5e40d5d3658b" Jul 06 19:21:04.455266 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.455237 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:21:04.455454 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:04.455364 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4s4w" podUID="8d5c45b9-a5f0-421d-bf44-59461dfd8fd0" Jul 06 19:21:04.754380 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.754310 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-211.ec2.internal" event="NodeReady" Jul 06 19:21:04.754755 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.754398 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:21:04.827263 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.827227 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5xtc8"] Jul 06 19:21:04.855248 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.855226 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8dsxn"] Jul 06 19:21:04.855387 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.855359 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5xtc8" Jul 06 19:21:04.858451 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.858431 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jkg7b\"" Jul 06 19:21:04.858561 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.858431 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:21:04.858857 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.858841 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:21:04.858952 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.858934 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:21:04.865568 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.865544 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5xtc8"] Jul 06 19:21:04.865725 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.865703 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.865782 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.865744 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8dsxn"] Jul 06 19:21:04.871195 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.871022 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:21:04.871676 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.871655 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:21:04.871770 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.871717 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:21:04.871839 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.871789 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2lnlz\"" Jul 06 19:21:04.871926 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.871907 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:21:04.882674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.882654 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73892e7-6ba0-486b-ae84-98b711e76ed2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.882768 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.882691 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b73892e7-6ba0-486b-ae84-98b711e76ed2-data-volume\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.882768 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.882717 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7vk\" (UniqueName: \"kubernetes.io/projected/b73892e7-6ba0-486b-ae84-98b711e76ed2-kube-api-access-gb7vk\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.882768 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.882762 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a55cefa-2f6c-45a7-99aa-d2d14317c804-cert\") pod \"ingress-canary-5xtc8\" (UID: \"3a55cefa-2f6c-45a7-99aa-d2d14317c804\") " pod="openshift-ingress-canary/ingress-canary-5xtc8" Jul 06 19:21:04.882874 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.882797 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7fk\" (UniqueName: \"kubernetes.io/projected/3a55cefa-2f6c-45a7-99aa-d2d14317c804-kube-api-access-rd7fk\") pod \"ingress-canary-5xtc8\" (UID: \"3a55cefa-2f6c-45a7-99aa-d2d14317c804\") " pod="openshift-ingress-canary/ingress-canary-5xtc8" Jul 06 19:21:04.882874 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.882811 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b73892e7-6ba0-486b-ae84-98b711e76ed2-crio-socket\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.882874 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.882845 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73892e7-6ba0-486b-ae84-98b711e76ed2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.923236 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.923212 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6flwj"] Jul 06 19:21:04.937453 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.937429 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6flwj"] Jul 06 19:21:04.937557 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.937533 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:04.939904 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.939887 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:21:04.940124 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.940096 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r5ggw\"" Jul 06 19:21:04.940124 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.940108 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:21:04.984061 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21aa5d6c-ede2-462b-b876-c95e1b1b4287-tmp-dir\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:04.984145 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7fk\" (UniqueName: \"kubernetes.io/projected/3a55cefa-2f6c-45a7-99aa-d2d14317c804-kube-api-access-rd7fk\") pod \"ingress-canary-5xtc8\" (UID: \"3a55cefa-2f6c-45a7-99aa-d2d14317c804\") " pod="openshift-ingress-canary/ingress-canary-5xtc8" Jul 06 19:21:04.984145 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b73892e7-6ba0-486b-ae84-98b711e76ed2-crio-socket\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.984222 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73892e7-6ba0-486b-ae84-98b711e76ed2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.984222 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984194 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21aa5d6c-ede2-462b-b876-c95e1b1b4287-config-volume\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:04.984328 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984227 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73892e7-6ba0-486b-ae84-98b711e76ed2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.984328 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b73892e7-6ba0-486b-ae84-98b711e76ed2-data-volume\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.984328 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984293 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21aa5d6c-ede2-462b-b876-c95e1b1b4287-metrics-tls\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:04.984328 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984298 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b73892e7-6ba0-486b-ae84-98b711e76ed2-crio-socket\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.984328 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984317 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cncg7\" (UniqueName: \"kubernetes.io/projected/21aa5d6c-ede2-462b-b876-c95e1b1b4287-kube-api-access-cncg7\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:04.984574 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984422 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7vk\" (UniqueName: \"kubernetes.io/projected/b73892e7-6ba0-486b-ae84-98b711e76ed2-kube-api-access-gb7vk\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.984574 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984463 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a55cefa-2f6c-45a7-99aa-d2d14317c804-cert\") pod \"ingress-canary-5xtc8\" (UID: \"3a55cefa-2f6c-45a7-99aa-d2d14317c804\") " pod="openshift-ingress-canary/ingress-canary-5xtc8" Jul 06 19:21:04.984691 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984635 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b73892e7-6ba0-486b-ae84-98b711e76ed2-data-volume\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.984880 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.984860 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73892e7-6ba0-486b-ae84-98b711e76ed2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.988047 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.988023 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a55cefa-2f6c-45a7-99aa-d2d14317c804-cert\") pod \"ingress-canary-5xtc8\" (UID: \"3a55cefa-2f6c-45a7-99aa-d2d14317c804\") " pod="openshift-ingress-canary/ingress-canary-5xtc8" Jul 06 19:21:04.988047 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.988036 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73892e7-6ba0-486b-ae84-98b711e76ed2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.993236 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.993209 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7vk\" (UniqueName: \"kubernetes.io/projected/b73892e7-6ba0-486b-ae84-98b711e76ed2-kube-api-access-gb7vk\") pod \"insights-runtime-extractor-8dsxn\" (UID: \"b73892e7-6ba0-486b-ae84-98b711e76ed2\") " pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:04.993967 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:04.993938 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7fk\" (UniqueName: \"kubernetes.io/projected/3a55cefa-2f6c-45a7-99aa-d2d14317c804-kube-api-access-rd7fk\") pod \"ingress-canary-5xtc8\" (UID: \"3a55cefa-2f6c-45a7-99aa-d2d14317c804\") " pod="openshift-ingress-canary/ingress-canary-5xtc8" Jul 06 19:21:05.085626 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.085601 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21aa5d6c-ede2-462b-b876-c95e1b1b4287-tmp-dir\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:05.085712 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.085656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21aa5d6c-ede2-462b-b876-c95e1b1b4287-config-volume\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:05.085777 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.085712 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21aa5d6c-ede2-462b-b876-c95e1b1b4287-metrics-tls\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:05.085777 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.085736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cncg7\" (UniqueName: \"kubernetes.io/projected/21aa5d6c-ede2-462b-b876-c95e1b1b4287-kube-api-access-cncg7\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:05.085907 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.085884 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21aa5d6c-ede2-462b-b876-c95e1b1b4287-tmp-dir\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:05.086212 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.086195 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21aa5d6c-ede2-462b-b876-c95e1b1b4287-config-volume\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:05.087670 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.087655 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21aa5d6c-ede2-462b-b876-c95e1b1b4287-metrics-tls\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:05.094765 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.094736 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cncg7\" (UniqueName: \"kubernetes.io/projected/21aa5d6c-ede2-462b-b876-c95e1b1b4287-kube-api-access-cncg7\") pod \"dns-default-6flwj\" (UID: \"21aa5d6c-ede2-462b-b876-c95e1b1b4287\") " pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:05.165661 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.165642 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5xtc8" Jul 06 19:21:05.173808 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.173792 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8dsxn" Jul 06 19:21:05.246619 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.246557 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:05.387328 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.387300 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5xtc8"] Jul 06 19:21:05.390931 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:05.390892 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a55cefa_2f6c_45a7_99aa_d2d14317c804.slice/crio-f11985447964d37308b1bab9cf7b6c533c0953832701005b3d7afbc51f17aafc WatchSource:0}: Error finding container f11985447964d37308b1bab9cf7b6c533c0953832701005b3d7afbc51f17aafc: Status 404 returned error can't find the container with id f11985447964d37308b1bab9cf7b6c533c0953832701005b3d7afbc51f17aafc Jul 06 19:21:05.402428 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.402394 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8dsxn"] Jul 06 19:21:05.405798 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:05.405776 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73892e7_6ba0_486b_ae84_98b711e76ed2.slice/crio-b1cac4d1756bac4cc04f1e97c99fcdc2845f2ba1073877b8339376ed6187b936 WatchSource:0}: Error finding container b1cac4d1756bac4cc04f1e97c99fcdc2845f2ba1073877b8339376ed6187b936: Status 404 returned error can't find the container with id b1cac4d1756bac4cc04f1e97c99fcdc2845f2ba1073877b8339376ed6187b936 Jul 06 19:21:05.418255 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.418234 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6flwj"] Jul 06 19:21:05.421133 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:05.421116 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21aa5d6c_ede2_462b_b876_c95e1b1b4287.slice/crio-56b0abb47530affebf363c6fc7ca67998aeab22f41b74fe7a944fc68d9e82aeb WatchSource:0}: Error finding container 56b0abb47530affebf363c6fc7ca67998aeab22f41b74fe7a944fc68d9e82aeb: Status 404 returned error can't find the container with id 56b0abb47530affebf363c6fc7ca67998aeab22f41b74fe7a944fc68d9e82aeb Jul 06 19:21:05.455746 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.455717 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:21:05.455913 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.455901 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:21:05.458458 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.458440 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:21:05.458629 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.458460 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c88qg\"" Jul 06 19:21:05.458629 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.458558 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:21:05.836644 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.836614 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8dsxn" event={"ID":"b73892e7-6ba0-486b-ae84-98b711e76ed2","Type":"ContainerStarted","Data":"b1cac4d1756bac4cc04f1e97c99fcdc2845f2ba1073877b8339376ed6187b936"} Jul 06 19:21:05.837715 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.837688 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6flwj" event={"ID":"21aa5d6c-ede2-462b-b876-c95e1b1b4287","Type":"ContainerStarted","Data":"56b0abb47530affebf363c6fc7ca67998aeab22f41b74fe7a944fc68d9e82aeb"} Jul 06 19:21:05.838734 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:05.838712 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5xtc8" event={"ID":"3a55cefa-2f6c-45a7-99aa-d2d14317c804","Type":"ContainerStarted","Data":"f11985447964d37308b1bab9cf7b6c533c0953832701005b3d7afbc51f17aafc"} Jul 06 19:21:06.294158 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.293736 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk"] Jul 06 19:21:06.318218 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.318166 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk"] Jul 06 19:21:06.318381 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.318349 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.321474 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.321383 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 19:21:06.321613 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.321579 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:21:06.322732 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.322700 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-v5k6n\"" Jul 06 19:21:06.322930 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.322914 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 19:21:06.323187 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.323171 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:21:06.323458 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.323443 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:21:06.396861 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.396822 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnk4\" (UniqueName: \"kubernetes.io/projected/4b87c79d-e82e-419a-8145-3d6c0e562304-kube-api-access-qqnk4\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.397031 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.396876 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b87c79d-e82e-419a-8145-3d6c0e562304-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.397031 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.396918 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b87c79d-e82e-419a-8145-3d6c0e562304-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.397031 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.396996 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b87c79d-e82e-419a-8145-3d6c0e562304-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.456408 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.455449 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:21:06.459185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.458815 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r9p6l\"" Jul 06 19:21:06.459185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.458821 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:21:06.459185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.458877 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:21:06.497627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.497597 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b87c79d-e82e-419a-8145-3d6c0e562304-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.497781 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.497679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnk4\" (UniqueName: \"kubernetes.io/projected/4b87c79d-e82e-419a-8145-3d6c0e562304-kube-api-access-qqnk4\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.497781 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.497719 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b87c79d-e82e-419a-8145-3d6c0e562304-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.497781 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.497748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b87c79d-e82e-419a-8145-3d6c0e562304-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.498546 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.498526 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b87c79d-e82e-419a-8145-3d6c0e562304-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.498714 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:06.498612 2571 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 06 19:21:06.498714 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:06.498653 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b87c79d-e82e-419a-8145-3d6c0e562304-prometheus-operator-tls podName:4b87c79d-e82e-419a-8145-3d6c0e562304 nodeName:}" failed. No retries permitted until 2026-07-06 19:21:06.998641048 +0000 UTC m=+54.058987953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4b87c79d-e82e-419a-8145-3d6c0e562304-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-25qwk" (UID: "4b87c79d-e82e-419a-8145-3d6c0e562304") : secret "prometheus-operator-tls" not found Jul 06 19:21:06.508001 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.506767 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b87c79d-e82e-419a-8145-3d6c0e562304-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.517365 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.517318 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnk4\" (UniqueName: \"kubernetes.io/projected/4b87c79d-e82e-419a-8145-3d6c0e562304-kube-api-access-qqnk4\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:06.754529 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.754451 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d5b6bff5-zzwm6"] Jul 06 19:21:06.786654 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.786610 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5b6bff5-zzwm6"] Jul 06 19:21:06.786811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.786739 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.791241 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.791023 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:21:06.791241 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.791114 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:21:06.791241 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.791023 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:21:06.793000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.792161 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:21:06.793000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.792179 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j7sv8\"" Jul 06 19:21:06.793000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.792243 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:21:06.793000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.792179 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:21:06.793000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.792431 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:21:06.800488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.800055 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-oauth-config\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.800488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.800133 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-console-config\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.800488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.800163 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2hr\" (UniqueName: \"kubernetes.io/projected/e7909354-3757-455a-9314-c734822f4fd0-kube-api-access-2m2hr\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.800488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.800206 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-serving-cert\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.800488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.800235 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-service-ca\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.800488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.800328 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-oauth-serving-cert\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.900883 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.900843 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-serving-cert\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.901316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.900889 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-service-ca\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.901316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.900946 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-oauth-serving-cert\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.901316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.900988 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-oauth-config\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.901316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.901029 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-console-config\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.901316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.901045 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2hr\" (UniqueName: \"kubernetes.io/projected/e7909354-3757-455a-9314-c734822f4fd0-kube-api-access-2m2hr\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.903882 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.902352 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-oauth-serving-cert\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.903882 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.902825 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-service-ca\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.903882 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.903841 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-console-config\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.904774 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.904706 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-serving-cert\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.912882 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.912838 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-oauth-config\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:06.915859 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:06.915800 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2hr\" (UniqueName: \"kubernetes.io/projected/e7909354-3757-455a-9314-c734822f4fd0-kube-api-access-2m2hr\") pod \"console-7d5b6bff5-zzwm6\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:07.002848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:07.002399 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b87c79d-e82e-419a-8145-3d6c0e562304-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:07.005503 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:07.005422 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4b87c79d-e82e-419a-8145-3d6c0e562304-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-25qwk\" (UID: \"4b87c79d-e82e-419a-8145-3d6c0e562304\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:07.100607 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:07.100572 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:07.235253 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:07.235035 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" Jul 06 19:21:07.277117 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:07.277082 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5b6bff5-zzwm6"] Jul 06 19:21:07.281024 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:07.280991 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7909354_3757_455a_9314_c734822f4fd0.slice/crio-0a3ac444e0bbdab0893e8ad58b798871fab0fb1c45c445e3d2ea888abe27361c WatchSource:0}: Error finding container 0a3ac444e0bbdab0893e8ad58b798871fab0fb1c45c445e3d2ea888abe27361c: Status 404 returned error can't find the container with id 0a3ac444e0bbdab0893e8ad58b798871fab0fb1c45c445e3d2ea888abe27361c Jul 06 19:21:07.409858 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:07.409825 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk"] Jul 06 19:21:07.847583 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:07.847544 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5b6bff5-zzwm6" event={"ID":"e7909354-3757-455a-9314-c734822f4fd0","Type":"ContainerStarted","Data":"0a3ac444e0bbdab0893e8ad58b798871fab0fb1c45c445e3d2ea888abe27361c"} Jul 06 19:21:07.949892 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:07.949854 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b87c79d_e82e_419a_8145_3d6c0e562304.slice/crio-932021ab935f150b7e093421dec1b39ef81e202d613d8f1c8fd93ace020fe450 WatchSource:0}: Error finding container 932021ab935f150b7e093421dec1b39ef81e202d613d8f1c8fd93ace020fe450: Status 404 returned error can't find the container with id 932021ab935f150b7e093421dec1b39ef81e202d613d8f1c8fd93ace020fe450 Jul 06 19:21:08.851983 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:08.851941 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" event={"ID":"4b87c79d-e82e-419a-8145-3d6c0e562304","Type":"ContainerStarted","Data":"932021ab935f150b7e093421dec1b39ef81e202d613d8f1c8fd93ace020fe450"} Jul 06 19:21:13.867092 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.867055 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" event={"ID":"4b87c79d-e82e-419a-8145-3d6c0e562304","Type":"ContainerStarted","Data":"4f918f3a6ce55fca3d2d2174ccbea51a7de4c1a2920ab2352685a2857fe2daa1"} Jul 06 19:21:13.867527 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.867105 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" event={"ID":"4b87c79d-e82e-419a-8145-3d6c0e562304","Type":"ContainerStarted","Data":"6e4ef47b52f4f4a6463497f9d666f0f996892a92419e6528c140cfb311be1271"} Jul 06 19:21:13.868628 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.868601 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5b6bff5-zzwm6" event={"ID":"e7909354-3757-455a-9314-c734822f4fd0","Type":"ContainerStarted","Data":"21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41"} Jul 06 19:21:13.870325 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.870303 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8dsxn" event={"ID":"b73892e7-6ba0-486b-ae84-98b711e76ed2","Type":"ContainerStarted","Data":"d7605edb1a1e173de3d85e8d117adfd0053c446750d7bd2ce2f251960a83a187"} Jul 06 19:21:13.870455 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.870353 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8dsxn" event={"ID":"b73892e7-6ba0-486b-ae84-98b711e76ed2","Type":"ContainerStarted","Data":"c93365ad61b3636021aaf4da32d7d4052f7f86fde2af7c6e564bf62e5ccc0cc5"} Jul 06 19:21:13.872062 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.872037 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6flwj" event={"ID":"21aa5d6c-ede2-462b-b876-c95e1b1b4287","Type":"ContainerStarted","Data":"fafdc33ad99732c9fc9b2a7c343840929f9d93984bb8642e8b0f966ad01189e4"} Jul 06 19:21:13.872184 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.872084 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6flwj" event={"ID":"21aa5d6c-ede2-462b-b876-c95e1b1b4287","Type":"ContainerStarted","Data":"c9fc022608b7b15caa3eb6e9e4928a02357c9eee945df947f2875debe84c6ed7"} Jul 06 19:21:13.872184 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.872139 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:13.873397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.873378 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5xtc8" event={"ID":"3a55cefa-2f6c-45a7-99aa-d2d14317c804","Type":"ContainerStarted","Data":"86bbabb0d06f259420c984e0386a436202cb085dbb95deeffd0c672ef5871633"} Jul 06 19:21:13.886734 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.886688 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-25qwk" podStartSLOduration=2.922419049 podStartE2EDuration="7.886677258s" totalImagesPullingTime="4.964258209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:13.88575733 +0000 UTC m=+60.946104257" watchObservedRunningTime="2026-07-06 19:21:13.886677258 +0000 UTC m=+60.947024161" Jul 06 19:21:13.908079 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.908032 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6flwj" podStartSLOduration=2.764135536 podStartE2EDuration="9.908016938s" totalImagesPullingTime="7.143881402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:13.907835075 +0000 UTC m=+60.968182001" watchObservedRunningTime="2026-07-06 19:21:13.908016938 +0000 UTC m=+60.968363856" Jul 06 19:21:13.927004 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.926964 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d5b6bff5-zzwm6" podStartSLOduration=2.289425945 podStartE2EDuration="7.926951396s" totalImagesPullingTime="5.637525451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:13.926317931 +0000 UTC m=+60.986664859" watchObservedRunningTime="2026-07-06 19:21:13.926951396 +0000 UTC m=+60.987298299" Jul 06 19:21:13.944524 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:13.944485 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5xtc8" podStartSLOduration=2.426354465 podStartE2EDuration="9.944473335s" totalImagesPullingTime="7.51811887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:13.944382483 +0000 UTC m=+61.004729420" watchObservedRunningTime="2026-07-06 19:21:13.944473335 +0000 UTC m=+61.004820260" Jul 06 19:21:15.646824 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.646787 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6"] Jul 06 19:21:15.652949 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.652921 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.655574 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.655546 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:21:15.655809 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.655775 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 19:21:15.655809 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.655798 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-c9gmv\"" Jul 06 19:21:15.663171 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.663018 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6"] Jul 06 19:21:15.666661 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.666637 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-94jtt"] Jul 06 19:21:15.670710 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.670683 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87"] Jul 06 19:21:15.670816 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.670795 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.673097 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.673080 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.678915 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.678676 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87"] Jul 06 19:21:15.680098 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.680052 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:21:15.680241 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.680223 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:21:15.682052 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.682032 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:21:15.682164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.682139 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5c8v4\"" Jul 06 19:21:15.682224 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.682168 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 19:21:15.682297 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.682228 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:21:15.682561 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.682542 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8s9tm\"" Jul 06 19:21:15.682717 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.682691 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 19:21:15.773552 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773518 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce527849-225c-4e19-ae07-7e40bc516e3c-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.773739 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw6q8\" (UniqueName: \"kubernetes.io/projected/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-kube-api-access-rw6q8\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.773739 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8434d9ae-3571-4d23-873c-42ea6176d06e-metrics-client-ca\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.773739 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773653 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.773739 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773674 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-textfile\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.773923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773738 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce527849-225c-4e19-ae07-7e40bc516e3c-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.773923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773763 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.773923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773800 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.773923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773826 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.773923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773850 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.774148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773934 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.774148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.773974 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8434d9ae-3571-4d23-873c-42ea6176d06e-sys\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.774148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.774014 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblfw\" (UniqueName: \"kubernetes.io/projected/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-api-access-gblfw\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.774148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.774041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.774148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.774127 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-tls\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.774360 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.774210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-accelerators-collector-config\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.774360 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.774250 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdfqv\" (UniqueName: \"kubernetes.io/projected/8434d9ae-3571-4d23-873c-42ea6176d06e-kube-api-access-zdfqv\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.774360 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.774316 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-wtmp\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.774479 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.774357 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8434d9ae-3571-4d23-873c-42ea6176d06e-root\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.874886 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.874847 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.875060 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.874893 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8434d9ae-3571-4d23-873c-42ea6176d06e-sys\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875060 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.874928 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblfw\" (UniqueName: \"kubernetes.io/projected/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-api-access-gblfw\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.875060 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.874954 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.875060 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.874985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-tls\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875060 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:15.875003 2571 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 19:21:15.875060 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875015 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-accelerators-collector-config\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875060 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875046 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdfqv\" (UniqueName: \"kubernetes.io/projected/8434d9ae-3571-4d23-873c-42ea6176d06e-kube-api-access-zdfqv\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:15.875079 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-openshift-state-metrics-tls podName:5eefc10c-741b-4dad-b05b-0ac34b17d3c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:21:16.375058848 +0000 UTC m=+63.435405776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-8tqj6" (UID: "5eefc10c-741b-4dad-b05b-0ac34b17d3c1") : secret "openshift-state-metrics-tls" not found Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875097 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-wtmp\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875136 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8434d9ae-3571-4d23-873c-42ea6176d06e-root\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce527849-225c-4e19-ae07-7e40bc516e3c-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875192 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw6q8\" (UniqueName: \"kubernetes.io/projected/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-kube-api-access-rw6q8\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875213 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8434d9ae-3571-4d23-873c-42ea6176d06e-metrics-client-ca\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875307 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-textfile\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875354 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce527849-225c-4e19-ae07-7e40bc516e3c-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875381 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8434d9ae-3571-4d23-873c-42ea6176d06e-sys\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875382 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875425 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875450 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.875485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875475 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.876180 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875691 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-wtmp\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.876180 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.875742 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8434d9ae-3571-4d23-873c-42ea6176d06e-root\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.876292 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.876245 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.876403 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.876366 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce527849-225c-4e19-ae07-7e40bc516e3c-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.877042 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.877017 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-accelerators-collector-config\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.877042 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.877036 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8434d9ae-3571-4d23-873c-42ea6176d06e-metrics-client-ca\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.877637 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.877610 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.877730 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.877675 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce527849-225c-4e19-ae07-7e40bc516e3c-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.877730 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.877678 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-textfile\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.877811 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:15.877768 2571 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 06 19:21:15.877857 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:15.877831 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-tls podName:ce527849-225c-4e19-ae07-7e40bc516e3c nodeName:}" failed. No retries permitted until 2026-07-06 19:21:16.377811461 +0000 UTC m=+63.438158369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-vbg87" (UID: "ce527849-225c-4e19-ae07-7e40bc516e3c") : secret "kube-state-metrics-tls" not found Jul 06 19:21:15.879948 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.879905 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-tls\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.880048 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.880012 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.880048 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.880012 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.882374 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.882335 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8434d9ae-3571-4d23-873c-42ea6176d06e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.887000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.886952 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdfqv\" (UniqueName: \"kubernetes.io/projected/8434d9ae-3571-4d23-873c-42ea6176d06e-kube-api-access-zdfqv\") pod \"node-exporter-94jtt\" (UID: \"8434d9ae-3571-4d23-873c-42ea6176d06e\") " pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:15.887604 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.887427 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblfw\" (UniqueName: \"kubernetes.io/projected/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-api-access-gblfw\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:15.887984 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.887953 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw6q8\" (UniqueName: \"kubernetes.io/projected/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-kube-api-access-rw6q8\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:15.984077 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:15.983355 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-94jtt" Jul 06 19:21:16.161901 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:16.161870 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8434d9ae_3571_4d23_873c_42ea6176d06e.slice/crio-2b082a7a041cdc0eadce8afdc6764a223295d91c746b9db5bfb10379509c5dc7 WatchSource:0}: Error finding container 2b082a7a041cdc0eadce8afdc6764a223295d91c746b9db5bfb10379509c5dc7: Status 404 returned error can't find the container with id 2b082a7a041cdc0eadce8afdc6764a223295d91c746b9db5bfb10379509c5dc7 Jul 06 19:21:16.335309 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.335262 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d4f78764c-9xm54"] Jul 06 19:21:16.340282 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.340257 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.348891 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.348866 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:21:16.353652 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.353619 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d4f78764c-9xm54"] Jul 06 19:21:16.380360 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.380336 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-console-config\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.380483 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.380366 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snsdq\" (UniqueName: \"kubernetes.io/projected/5666d89c-e698-4a92-8c1f-401596cbda92-kube-api-access-snsdq\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.380483 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.380398 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:16.380483 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.380438 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-oauth-serving-cert\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.380648 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.380489 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:16.380648 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.380540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-oauth-config\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.380648 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.380628 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-service-ca\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.380752 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.380657 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-trusted-ca-bundle\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.380752 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.380685 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-serving-cert\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.382785 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.382763 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce527849-225c-4e19-ae07-7e40bc516e3c-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-vbg87\" (UID: \"ce527849-225c-4e19-ae07-7e40bc516e3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:16.382869 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.382788 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eefc10c-741b-4dad-b05b-0ac34b17d3c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-8tqj6\" (UID: \"5eefc10c-741b-4dad-b05b-0ac34b17d3c1\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:16.480952 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.480927 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-serving-cert\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.481063 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.480973 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-console-config\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.481063 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.480991 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snsdq\" (UniqueName: \"kubernetes.io/projected/5666d89c-e698-4a92-8c1f-401596cbda92-kube-api-access-snsdq\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.481063 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.481019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-oauth-serving-cert\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.481063 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.481058 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-oauth-config\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.481243 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.481075 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-service-ca\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.481243 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.481163 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-trusted-ca-bundle\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.481972 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.481876 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-service-ca\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.481972 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.481892 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-oauth-serving-cert\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.482155 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.482025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-console-config\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.482621 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.482601 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-trusted-ca-bundle\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.483462 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.483436 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-serving-cert\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.483686 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.483668 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-oauth-config\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.489233 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.489214 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snsdq\" (UniqueName: \"kubernetes.io/projected/5666d89c-e698-4a92-8c1f-401596cbda92-kube-api-access-snsdq\") pod \"console-6d4f78764c-9xm54\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.569751 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.569731 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" Jul 06 19:21:16.589642 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.589625 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" Jul 06 19:21:16.650020 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.649786 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:16.745846 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.745757 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6"] Jul 06 19:21:16.750120 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:16.750092 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eefc10c_741b_4dad_b05b_0ac34b17d3c1.slice/crio-ca951b112b56ba3c1ec4bea1a2fc54576162f335d4b6bd746148e5eec12923f1 WatchSource:0}: Error finding container ca951b112b56ba3c1ec4bea1a2fc54576162f335d4b6bd746148e5eec12923f1: Status 404 returned error can't find the container with id ca951b112b56ba3c1ec4bea1a2fc54576162f335d4b6bd746148e5eec12923f1 Jul 06 19:21:16.782331 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.781123 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87"] Jul 06 19:21:16.782331 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.781200 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:21:16.797296 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:16.797243 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce527849_225c_4e19_ae07_7e40bc516e3c.slice/crio-d633cbbd14f6e39697f00eca4d63be103db12b360ef658bf01730f7c8976b317 WatchSource:0}: Error finding container d633cbbd14f6e39697f00eca4d63be103db12b360ef658bf01730f7c8976b317: Status 404 returned error can't find the container with id d633cbbd14f6e39697f00eca4d63be103db12b360ef658bf01730f7c8976b317 Jul 06 19:21:16.798095 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.798076 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.804308 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.803682 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.804486 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.805085 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xc2t" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.805521 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.806012 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.807216 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.807579 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.807688 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s67qf\"" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.807939 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.808407 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 19:21:16.809509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.808731 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 19:21:16.813290 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.810777 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:21:16.839782 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.839757 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d4f78764c-9xm54"] Jul 06 19:21:16.884142 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.884123 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-out\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.884357 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.884336 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.884554 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.884495 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.884686 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.884671 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-web-config\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.884804 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.884791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.884949 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.884934 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.885320 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.885303 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-volume\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.885474 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.885461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.886249 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.886233 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.886362 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.886339 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqzrs\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-kube-api-access-vqzrs\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.886424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.886374 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.886424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.886412 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.886524 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.886479 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.890469 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.890440 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" event={"ID":"5eefc10c-741b-4dad-b05b-0ac34b17d3c1","Type":"ContainerStarted","Data":"676fae66ed1e592efb657ba5c09e326665e74042a3fdb0c45ddaf42c64233d03"} Jul 06 19:21:16.890568 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.890503 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" event={"ID":"5eefc10c-741b-4dad-b05b-0ac34b17d3c1","Type":"ContainerStarted","Data":"ca951b112b56ba3c1ec4bea1a2fc54576162f335d4b6bd746148e5eec12923f1"} Jul 06 19:21:16.891923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.891861 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" event={"ID":"ce527849-225c-4e19-ae07-7e40bc516e3c","Type":"ContainerStarted","Data":"d633cbbd14f6e39697f00eca4d63be103db12b360ef658bf01730f7c8976b317"} Jul 06 19:21:16.893195 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.893140 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94jtt" event={"ID":"8434d9ae-3571-4d23-873c-42ea6176d06e","Type":"ContainerStarted","Data":"2b082a7a041cdc0eadce8afdc6764a223295d91c746b9db5bfb10379509c5dc7"} Jul 06 19:21:16.895794 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.895756 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8dsxn" event={"ID":"b73892e7-6ba0-486b-ae84-98b711e76ed2","Type":"ContainerStarted","Data":"ecf86ba9c12579828011312d2c80c76df5974f5e8f5756f82604c3eb2f4af9c9"} Jul 06 19:21:16.900515 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.900496 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4f78764c-9xm54" event={"ID":"5666d89c-e698-4a92-8c1f-401596cbda92","Type":"ContainerStarted","Data":"29e27635ca7961aa71b63a507ed2df6be149635b81bc234ed68f0f05ba0ef894"} Jul 06 19:21:16.917400 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.916708 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8dsxn" podStartSLOduration=2.333000694 podStartE2EDuration="12.916691222s" totalImagesPullingTime="10.583690528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:21:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:16.915922181 +0000 UTC m=+63.976269108" watchObservedRunningTime="2026-07-06 19:21:16.916691222 +0000 UTC m=+63.977038149" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987224 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-out\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987319 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987354 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-web-config\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987423 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987465 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-volume\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987491 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987548 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqzrs\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-kube-api-access-vqzrs\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987582 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.987627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987608 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.988144 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.987663 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.988556 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:16.988379 2571 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 19:21:16.988556 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:16.988445 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-main-tls podName:9341c82a-df20-483b-ad3d-dc16ad3e4b49 nodeName:}" failed. No retries permitted until 2026-07-06 19:21:17.488425005 +0000 UTC m=+64.548771927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49") : secret "alertmanager-main-tls" not found Jul 06 19:21:16.989777 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.989531 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.989777 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.989722 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.989923 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.989848 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.991584 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.991539 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.992159 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.992092 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.992296 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.992149 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-volume\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.996170 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.995750 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.996170 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.995872 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-web-config\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.996170 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.996037 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-out\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.996459 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.996418 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.997072 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.997031 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:16.999213 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:16.999193 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqzrs\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-kube-api-access-vqzrs\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:17.101715 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.101629 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:17.101880 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.101749 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:17.103115 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.103089 2571 patch_prober.go:28] interesting pod/console-7d5b6bff5-zzwm6 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 06 19:21:17.103232 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.103135 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7d5b6bff5-zzwm6" podUID="e7909354-3757-455a-9314-c734822f4fd0" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 06 19:21:17.493150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.493075 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:17.495821 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.495793 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:17.724328 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.724304 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:21:17.799048 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.798291 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54f87764f-gc958"] Jul 06 19:21:17.818936 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.818838 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54f87764f-gc958"] Jul 06 19:21:17.819134 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.818985 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.822081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.822051 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 19:21:17.822388 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.822365 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 19:21:17.822561 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.822537 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 19:21:17.822636 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.822613 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 19:21:17.822703 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.822477 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 19:21:17.822703 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.822374 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hrtff\"" Jul 06 19:21:17.822951 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.822933 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2cupipaakc75s\"" Jul 06 19:21:17.897393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.897355 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-tls\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.897534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.897413 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.897534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.897461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.897534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.897486 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1e27927-a3fb-4e73-854c-0175744772a6-metrics-client-ca\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.897534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.897517 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.897758 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.897539 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfdn\" (UniqueName: \"kubernetes.io/projected/a1e27927-a3fb-4e73-854c-0175744772a6-kube-api-access-vnfdn\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.897758 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.897587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-grpc-tls\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.897758 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.897713 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.909697 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.909669 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" event={"ID":"5eefc10c-741b-4dad-b05b-0ac34b17d3c1","Type":"ContainerStarted","Data":"bef2df6aa37c3241052f8b9054c7ce53500f35d1ca7c820904dc9fd5b159ae8e"} Jul 06 19:21:17.912881 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.912471 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94jtt" event={"ID":"8434d9ae-3571-4d23-873c-42ea6176d06e","Type":"ContainerStarted","Data":"d07c4d18810f073b5eb9dc322000a38f6180c11b581547a7aefa1fb930bf416f"} Jul 06 19:21:17.913112 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.913077 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:21:17.914778 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.914719 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4f78764c-9xm54" event={"ID":"5666d89c-e698-4a92-8c1f-401596cbda92","Type":"ContainerStarted","Data":"e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d"} Jul 06 19:21:17.948328 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:17.948300 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9341c82a_df20_483b_ad3d_dc16ad3e4b49.slice/crio-d781374d479a769808a082508592ad39292ab6624b037b40fa984f6e737b19a4 WatchSource:0}: Error finding container d781374d479a769808a082508592ad39292ab6624b037b40fa984f6e737b19a4: Status 404 returned error can't find the container with id d781374d479a769808a082508592ad39292ab6624b037b40fa984f6e737b19a4 Jul 06 19:21:17.953939 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.953892 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d4f78764c-9xm54" podStartSLOduration=1.953873433 podStartE2EDuration="1.953873433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:17.952435205 +0000 UTC m=+65.012782129" watchObservedRunningTime="2026-07-06 19:21:17.953873433 +0000 UTC m=+65.014220340" Jul 06 19:21:17.998164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.998102 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-grpc-tls\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.998596 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.998564 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.998719 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.998700 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-tls\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.998789 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.998742 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.998887 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.998863 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.999088 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.999058 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1e27927-a3fb-4e73-854c-0175744772a6-metrics-client-ca\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.999186 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.999102 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:17.999186 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:17.999130 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfdn\" (UniqueName: \"kubernetes.io/projected/a1e27927-a3fb-4e73-854c-0175744772a6-kube-api-access-vnfdn\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.000442 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.000415 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1e27927-a3fb-4e73-854c-0175744772a6-metrics-client-ca\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.001505 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.001480 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-grpc-tls\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.001778 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.001750 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.001929 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.001911 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-tls\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.002058 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.002035 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.002126 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.002095 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.002266 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.002245 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a1e27927-a3fb-4e73-854c-0175744772a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.007682 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.007662 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfdn\" (UniqueName: \"kubernetes.io/projected/a1e27927-a3fb-4e73-854c-0175744772a6-kube-api-access-vnfdn\") pod \"thanos-querier-54f87764f-gc958\" (UID: \"a1e27927-a3fb-4e73-854c-0175744772a6\") " pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.132883 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.132672 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:18.305216 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.305181 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54f87764f-gc958"] Jul 06 19:21:18.557097 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:18.557011 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e27927_a3fb_4e73_854c_0175744772a6.slice/crio-0d38104b2525223ddf8daac0d91a382cfe5236027c62dd38de5e90aa9fa1a1ce WatchSource:0}: Error finding container 0d38104b2525223ddf8daac0d91a382cfe5236027c62dd38de5e90aa9fa1a1ce: Status 404 returned error can't find the container with id 0d38104b2525223ddf8daac0d91a382cfe5236027c62dd38de5e90aa9fa1a1ce Jul 06 19:21:18.923855 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.923030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" event={"ID":"5eefc10c-741b-4dad-b05b-0ac34b17d3c1","Type":"ContainerStarted","Data":"a6a5b9bcb3a0084037fc99d2b78b118060a2f9d49d74ded501db76de80931f59"} Jul 06 19:21:18.926697 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.926663 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" event={"ID":"ce527849-225c-4e19-ae07-7e40bc516e3c","Type":"ContainerStarted","Data":"c8d695b2a01de8c90425b7b0440246c1b727bf757bf677b341d811908bf8227c"} Jul 06 19:21:18.933868 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.933264 2571 generic.go:356] "Generic (PLEG): container finished" podID="8434d9ae-3571-4d23-873c-42ea6176d06e" containerID="d07c4d18810f073b5eb9dc322000a38f6180c11b581547a7aefa1fb930bf416f" exitCode=0 Jul 06 19:21:18.933868 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.933362 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94jtt" event={"ID":"8434d9ae-3571-4d23-873c-42ea6176d06e","Type":"ContainerDied","Data":"d07c4d18810f073b5eb9dc322000a38f6180c11b581547a7aefa1fb930bf416f"} Jul 06 19:21:18.936727 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.936655 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" event={"ID":"a1e27927-a3fb-4e73-854c-0175744772a6","Type":"ContainerStarted","Data":"0d38104b2525223ddf8daac0d91a382cfe5236027c62dd38de5e90aa9fa1a1ce"} Jul 06 19:21:18.956332 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.954090 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerStarted","Data":"d781374d479a769808a082508592ad39292ab6624b037b40fa984f6e737b19a4"} Jul 06 19:21:18.997645 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:18.997593 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-8tqj6" podStartSLOduration=2.177214499 podStartE2EDuration="3.99754461s" totalImagesPullingTime="1.820330111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:18.953821289 +0000 UTC m=+66.014168227" watchObservedRunningTime="2026-07-06 19:21:18.99754461 +0000 UTC m=+66.057891537" Jul 06 19:21:19.109719 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.109661 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:21:19.112180 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.112157 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:21:19.130293 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.127690 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44d4a5a8-5d12-483d-939c-5e40d5d3658b-original-pull-secret\") pod \"global-pull-secret-syncer-8rchd\" (UID: \"44d4a5a8-5d12-483d-939c-5e40d5d3658b\") " pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:21:19.210895 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.210850 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:21:19.211047 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.210921 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:21:19.213612 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.213308 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:21:19.213612 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.213506 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:21:19.223771 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.223749 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:21:19.224710 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.224672 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6-metrics-certs\") pod \"network-metrics-daemon-gdvls\" (UID: \"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6\") " pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:21:19.234745 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.234722 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnf6n\" (UniqueName: \"kubernetes.io/projected/8d5c45b9-a5f0-421d-bf44-59461dfd8fd0-kube-api-access-bnf6n\") pod \"network-check-target-h4s4w\" (UID: \"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0\") " pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:21:19.268112 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.268084 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c88qg\"" Jul 06 19:21:19.270031 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.270002 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rchd" Jul 06 19:21:19.277132 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.275962 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdvls" Jul 06 19:21:19.376257 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.376176 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r9p6l\"" Jul 06 19:21:19.384369 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.384342 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:21:19.655056 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.654985 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4s4w"] Jul 06 19:21:19.677903 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.677874 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gdvls"] Jul 06 19:21:19.698833 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.698807 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8rchd"] Jul 06 19:21:19.702325 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:19.702301 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5c45b9_a5f0_421d_bf44_59461dfd8fd0.slice/crio-f39685764383801e45b4de837e9fab2413e80d77fb94593ca8151c4f74bed678 WatchSource:0}: Error finding container f39685764383801e45b4de837e9fab2413e80d77fb94593ca8151c4f74bed678: Status 404 returned error can't find the container with id f39685764383801e45b4de837e9fab2413e80d77fb94593ca8151c4f74bed678 Jul 06 19:21:19.702574 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:19.702548 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7c170d_a1b6_4935_b8a3_cb4e4e26d8e6.slice/crio-0ddcba8b8071de2e7d52f0b8dfb796a559489290f7a90fa784c6d57d2751eaec WatchSource:0}: Error finding container 0ddcba8b8071de2e7d52f0b8dfb796a559489290f7a90fa784c6d57d2751eaec: Status 404 returned error can't find the container with id 0ddcba8b8071de2e7d52f0b8dfb796a559489290f7a90fa784c6d57d2751eaec Jul 06 19:21:19.703532 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:19.703475 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d4a5a8_5d12_483d_939c_5e40d5d3658b.slice/crio-1f934ed732c3e1e6b3f5f1d1200c7d150784bc4a1a44387fd0849edfb23a488e WatchSource:0}: Error finding container 1f934ed732c3e1e6b3f5f1d1200c7d150784bc4a1a44387fd0849edfb23a488e: Status 404 returned error can't find the container with id 1f934ed732c3e1e6b3f5f1d1200c7d150784bc4a1a44387fd0849edfb23a488e Jul 06 19:21:19.958636 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.958555 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp"] Jul 06 19:21:19.961394 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.961366 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4s4w" event={"ID":"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0","Type":"ContainerStarted","Data":"f39685764383801e45b4de837e9fab2413e80d77fb94593ca8151c4f74bed678"} Jul 06 19:21:19.961535 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.961478 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:19.961986 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.961939 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" event={"ID":"ce527849-225c-4e19-ae07-7e40bc516e3c","Type":"ContainerStarted","Data":"61e2d07bc5a827959f230205c22ee5543ff0d85bbe468880249447e2780a7160"} Jul 06 19:21:19.962097 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.962015 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" event={"ID":"ce527849-225c-4e19-ae07-7e40bc516e3c","Type":"ContainerStarted","Data":"bf0bcea4a4bead38045e42a742a96fb6b865ce2c294a0aafd759f3c39c0b2b09"} Jul 06 19:21:19.964504 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.964483 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94jtt" event={"ID":"8434d9ae-3571-4d23-873c-42ea6176d06e","Type":"ContainerStarted","Data":"9a31b8fbc6e18dc3882a3ac7fde316aa6fcdf53070adc0c2b967822f538862a5"} Jul 06 19:21:19.964628 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.964524 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94jtt" event={"ID":"8434d9ae-3571-4d23-873c-42ea6176d06e","Type":"ContainerStarted","Data":"f3ac4d1b9c409924e52c12964e3400fbb8836d2c76e1e51e85d2602d4f0885ad"} Jul 06 19:21:19.965082 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.965061 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 19:21:19.965521 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.965489 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9hl58e9v2k8j\"" Jul 06 19:21:19.965626 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.965499 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 19:21:19.965626 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.965607 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 19:21:19.965626 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.965500 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ktxnf\"" Jul 06 19:21:19.965768 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.965641 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 19:21:19.966511 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.966490 2571 generic.go:356] "Generic (PLEG): container finished" podID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerID="8f7ced3a04b6caf97e5d30c642b38af80e04c3ebc1a3ece2fc9f2567286455ac" exitCode=0 Jul 06 19:21:19.966584 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.966515 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerDied","Data":"8f7ced3a04b6caf97e5d30c642b38af80e04c3ebc1a3ece2fc9f2567286455ac"} Jul 06 19:21:19.967843 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.967825 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8rchd" event={"ID":"44d4a5a8-5d12-483d-939c-5e40d5d3658b","Type":"ContainerStarted","Data":"1f934ed732c3e1e6b3f5f1d1200c7d150784bc4a1a44387fd0849edfb23a488e"} Jul 06 19:21:19.969236 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.969212 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdvls" event={"ID":"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6","Type":"ContainerStarted","Data":"0ddcba8b8071de2e7d52f0b8dfb796a559489290f7a90fa784c6d57d2751eaec"} Jul 06 19:21:19.976678 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:19.976653 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp"] Jul 06 19:21:20.018306 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.018251 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-audit-log\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.019109 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.018676 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-metrics-server-audit-profiles\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.019109 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.019037 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99vk6\" (UniqueName: \"kubernetes.io/projected/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-kube-api-access-99vk6\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.021076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.019551 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.021076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.019605 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-client-ca-bundle\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.021076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.019757 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-secret-metrics-server-client-certs\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.021076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.019800 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-secret-metrics-server-tls\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.052095 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.052041 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-vbg87" podStartSLOduration=3.057085421 podStartE2EDuration="5.052024083s" totalImagesPullingTime="1.994938662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:20.050324241 +0000 UTC m=+67.110671172" watchObservedRunningTime="2026-07-06 19:21:20.052024083 +0000 UTC m=+67.112371009" Jul 06 19:21:20.080105 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.080051 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-94jtt" podStartSLOduration=3.456400956 podStartE2EDuration="5.080034851s" totalImagesPullingTime="1.495774296s" totalInitContainerRuntime="127.859599ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:20.079147857 +0000 UTC m=+67.139494782" watchObservedRunningTime="2026-07-06 19:21:20.080034851 +0000 UTC m=+67.140381777" Jul 06 19:21:20.121093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.121061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-audit-log\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.121093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.121095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-metrics-server-audit-profiles\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.121319 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.121112 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99vk6\" (UniqueName: \"kubernetes.io/projected/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-kube-api-access-99vk6\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.121319 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.121130 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.121319 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.121152 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-client-ca-bundle\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.121319 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.121189 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-secret-metrics-server-client-certs\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.121319 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.121214 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-secret-metrics-server-tls\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.121588 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.121527 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-audit-log\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.122041 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.121973 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.122230 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.122155 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-metrics-server-audit-profiles\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.123837 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.123804 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-client-ca-bundle\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.123961 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.123938 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-secret-metrics-server-tls\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.123961 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.123955 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-secret-metrics-server-client-certs\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.131148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.131127 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99vk6\" (UniqueName: \"kubernetes.io/projected/a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf-kube-api-access-99vk6\") pod \"metrics-server-7bf5f9c4b9-kvwtp\" (UID: \"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf\") " pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.272767 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.272734 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:20.441006 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.439529 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx"] Jul 06 19:21:20.442771 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.442140 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:20.444847 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.444826 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 19:21:20.445208 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.445191 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nnvcj\"" Jul 06 19:21:20.455095 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.454920 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx"] Jul 06 19:21:20.526116 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.525897 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08c932a1-c259-48fd-9f3b-65c1c51a536f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-cswwx\" (UID: \"08c932a1-c259-48fd-9f3b-65c1c51a536f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:20.526116 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.525963 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8ww\" (UniqueName: \"kubernetes.io/projected/08c932a1-c259-48fd-9f3b-65c1c51a536f-kube-api-access-tb8ww\") pod \"monitoring-plugin-d5dcff7cf-cswwx\" (UID: \"08c932a1-c259-48fd-9f3b-65c1c51a536f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:20.626601 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.626561 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08c932a1-c259-48fd-9f3b-65c1c51a536f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-cswwx\" (UID: \"08c932a1-c259-48fd-9f3b-65c1c51a536f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:20.626787 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.626630 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8ww\" (UniqueName: \"kubernetes.io/projected/08c932a1-c259-48fd-9f3b-65c1c51a536f-kube-api-access-tb8ww\") pod \"monitoring-plugin-d5dcff7cf-cswwx\" (UID: \"08c932a1-c259-48fd-9f3b-65c1c51a536f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:20.627418 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:20.627122 2571 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 19:21:20.627418 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:20.627201 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c932a1-c259-48fd-9f3b-65c1c51a536f-monitoring-plugin-cert podName:08c932a1-c259-48fd-9f3b-65c1c51a536f nodeName:}" failed. No retries permitted until 2026-07-06 19:21:21.127182531 +0000 UTC m=+68.187529435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/08c932a1-c259-48fd-9f3b-65c1c51a536f-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-cswwx" (UID: "08c932a1-c259-48fd-9f3b-65c1c51a536f") : secret "monitoring-plugin-cert" not found Jul 06 19:21:20.640813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:20.640773 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8ww\" (UniqueName: \"kubernetes.io/projected/08c932a1-c259-48fd-9f3b-65c1c51a536f-kube-api-access-tb8ww\") pod \"monitoring-plugin-d5dcff7cf-cswwx\" (UID: \"08c932a1-c259-48fd-9f3b-65c1c51a536f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:21.132104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:21.132067 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08c932a1-c259-48fd-9f3b-65c1c51a536f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-cswwx\" (UID: \"08c932a1-c259-48fd-9f3b-65c1c51a536f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:21.135072 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:21.135045 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08c932a1-c259-48fd-9f3b-65c1c51a536f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-cswwx\" (UID: \"08c932a1-c259-48fd-9f3b-65c1c51a536f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:21.357910 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:21.357685 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:21.440359 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:21.438456 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp"] Jul 06 19:21:21.448982 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:21.448951 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ef47f4_f4f0_48d1_a09b_89f6e6bbcacf.slice/crio-c055612fa2c1283a8ee9e79817a734f0317a8b72177ad5533b5a3237939f4266 WatchSource:0}: Error finding container c055612fa2c1283a8ee9e79817a734f0317a8b72177ad5533b5a3237939f4266: Status 404 returned error can't find the container with id c055612fa2c1283a8ee9e79817a734f0317a8b72177ad5533b5a3237939f4266 Jul 06 19:21:21.555981 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:21.555919 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx"] Jul 06 19:21:21.559749 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:21.559720 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c932a1_c259_48fd_9f3b_65c1c51a536f.slice/crio-6c0add20dbb4993f17ed4cd1bd2899465bf0b6065120833213d73e0c4b88254c WatchSource:0}: Error finding container 6c0add20dbb4993f17ed4cd1bd2899465bf0b6065120833213d73e0c4b88254c: Status 404 returned error can't find the container with id 6c0add20dbb4993f17ed4cd1bd2899465bf0b6065120833213d73e0c4b88254c Jul 06 19:21:21.999165 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:21.999129 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" event={"ID":"a1e27927-a3fb-4e73-854c-0175744772a6","Type":"ContainerStarted","Data":"e670c7bb78ec499b4f8cbed7a93526a955bd9cfc372abdcba60c2a29722cd541"} Jul 06 19:21:21.999768 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:21.999744 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" event={"ID":"a1e27927-a3fb-4e73-854c-0175744772a6","Type":"ContainerStarted","Data":"372d519db368ce5b598e65d29bf6576ef29955ac64a8b11cba03b4bcc7869445"} Jul 06 19:21:21.999945 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:21.999927 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" event={"ID":"a1e27927-a3fb-4e73-854c-0175744772a6","Type":"ContainerStarted","Data":"d695af49bb1b2bb847501cbf5fb6dae4645cbb71c4102af831ee58b9d48a15c6"} Jul 06 19:21:22.003425 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.003365 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" event={"ID":"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf","Type":"ContainerStarted","Data":"c055612fa2c1283a8ee9e79817a734f0317a8b72177ad5533b5a3237939f4266"} Jul 06 19:21:22.007548 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.007521 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdvls" event={"ID":"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6","Type":"ContainerStarted","Data":"cd9e6aff706ccbb9227fc43727791b1b194adfb91ec9fb37798ee0b9a689c49b"} Jul 06 19:21:22.007639 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.007570 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdvls" event={"ID":"3e7c170d-a1b6-4935-b8a3-cb4e4e26d8e6","Type":"ContainerStarted","Data":"3419008c8e6f6729eb96ff5a413511524de54f00177ead1b7c7d7aa866ff5cae"} Jul 06 19:21:22.010217 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.010196 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" event={"ID":"08c932a1-c259-48fd-9f3b-65c1c51a536f","Type":"ContainerStarted","Data":"6c0add20dbb4993f17ed4cd1bd2899465bf0b6065120833213d73e0c4b88254c"} Jul 06 19:21:22.024391 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.022004 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:21:22.026440 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.026414 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.028685 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.028606 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gdvls" podStartSLOduration=67.456315999 podStartE2EDuration="1m9.028589562s" totalImagesPullingTime="1.572273563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:22.028209175 +0000 UTC m=+69.088556112" watchObservedRunningTime="2026-07-06 19:21:22.028589562 +0000 UTC m=+69.088936493" Jul 06 19:21:22.029222 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.029193 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 19:21:22.029574 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.029559 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 19:21:22.029809 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.029794 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qbnwq\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.031676 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.031692 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.031772 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.031911 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.031926 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.031973 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.032202 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.032509 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.032559 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 19:21:22.032888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.032688 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 19:21:22.033441 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.032949 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3u80smmpugct5\"" Jul 06 19:21:22.035380 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.035044 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 19:21:22.038800 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.038778 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 19:21:22.049295 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.049220 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:21:22.153541 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.153491 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.154380 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154330 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-config-out\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154654 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-config\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154771 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154818 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nq8q\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-kube-api-access-7nq8q\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154847 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.154873 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.155035 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.155085 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-web-config\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.155133 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.155155 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.155183 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.156104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.155229 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.157001 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.155258 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.157001 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.155310 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.256734 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256140 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.256734 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-config-out\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.256734 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.256734 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256264 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257059 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256796 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257059 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256859 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257059 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256901 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257059 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256935 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-config\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257059 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256965 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257059 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.256994 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257059 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nq8q\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-kube-api-access-7nq8q\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257059 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257052 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257466 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257076 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257466 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257106 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257466 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257129 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-web-config\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257466 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257162 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257466 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257466 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257217 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257466 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257260 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257466 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257309 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.257466 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257342 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.259331 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.257261 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.259331 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.259011 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.261851 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.261618 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.262832 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.262380 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.262832 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.262754 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.267384 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.267289 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-config\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.269070 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.269017 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.270871 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.270419 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nq8q\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-kube-api-access-7nq8q\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.270871 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.270651 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.270871 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.270829 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.271625 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.271071 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.273226 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.273188 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-web-config\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.273226 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.273202 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.274601 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.274557 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.275345 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.275306 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.275521 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.275498 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.276252 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.276212 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-config-out\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.278489 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.278467 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.290953 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.290923 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.343173 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.343130 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:22.562744 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:22.562642 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:21:23.102733 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:23.102700 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c1f96f_e470_4e61_a477_263bd51a7840.slice/crio-d9348de7d37c35fa84549fcf4983b46feb384f8798703b691f78f79f95bf326c WatchSource:0}: Error finding container d9348de7d37c35fa84549fcf4983b46feb384f8798703b691f78f79f95bf326c: Status 404 returned error can't find the container with id d9348de7d37c35fa84549fcf4983b46feb384f8798703b691f78f79f95bf326c Jul 06 19:21:23.373076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.373003 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d5b6bff5-zzwm6"] Jul 06 19:21:23.411835 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.411801 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cd6c87777-zp7wc"] Jul 06 19:21:23.414733 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.414709 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.427612 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.427588 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd6c87777-zp7wc"] Jul 06 19:21:23.570928 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.570890 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-serving-cert\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.571102 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.570941 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-oauth-config\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.571102 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.571031 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-console-config\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.571102 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.571083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-service-ca\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.571310 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.571132 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzkj\" (UniqueName: \"kubernetes.io/projected/c32d39d7-ceae-487d-816a-65f8d80d2478-kube-api-access-rxzkj\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.571310 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.571163 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-oauth-serving-cert\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.571310 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.571287 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-trusted-ca-bundle\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.672257 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.672181 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-console-config\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.672257 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.672224 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-service-ca\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.672257 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.672253 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzkj\" (UniqueName: \"kubernetes.io/projected/c32d39d7-ceae-487d-816a-65f8d80d2478-kube-api-access-rxzkj\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.672533 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.672303 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-oauth-serving-cert\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.672533 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.672345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-trusted-ca-bundle\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.672533 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.672406 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-serving-cert\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.672533 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.672439 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-oauth-config\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.673053 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.673023 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-service-ca\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.673141 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.673023 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-console-config\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.673511 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.673490 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-trusted-ca-bundle\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.673627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.673597 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-oauth-serving-cert\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.675202 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.675179 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-oauth-config\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.675750 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.675730 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-serving-cert\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.681184 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.681167 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzkj\" (UniqueName: \"kubernetes.io/projected/c32d39d7-ceae-487d-816a-65f8d80d2478-kube-api-access-rxzkj\") pod \"console-5cd6c87777-zp7wc\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.727287 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.727247 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:23.881254 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:23.881221 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6flwj" Jul 06 19:21:24.021849 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:24.021809 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerStarted","Data":"d9348de7d37c35fa84549fcf4983b46feb384f8798703b691f78f79f95bf326c"} Jul 06 19:21:26.595924 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:26.595858 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd6c87777-zp7wc"] Jul 06 19:21:26.647450 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:21:26.647425 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32d39d7_ceae_487d_816a_65f8d80d2478.slice/crio-3ad05cf5235f77d4416eecd070cec71013a4409bf186c0f272c60da986a0f728 WatchSource:0}: Error finding container 3ad05cf5235f77d4416eecd070cec71013a4409bf186c0f272c60da986a0f728: Status 404 returned error can't find the container with id 3ad05cf5235f77d4416eecd070cec71013a4409bf186c0f272c60da986a0f728 Jul 06 19:21:26.650389 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:26.650368 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:26.650495 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:26.650420 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:26.657086 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:26.657064 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:27.042952 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.042921 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" event={"ID":"a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf","Type":"ContainerStarted","Data":"253fd47ead900f43094f2e9cc82447b507807dc9650104c3701c9f213ad329f0"} Jul 06 19:21:27.046945 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.046923 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerStarted","Data":"1ed08eb16375ef3f950c0d2887f598e232086c1ee2fdfa127aeb19519a03da14"} Jul 06 19:21:27.047052 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.046988 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerStarted","Data":"0eee0b12901158bd6c7b9ed5eba00dcb50667fb9bb554287a8a281d0139c2619"} Jul 06 19:21:27.047052 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.047030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerStarted","Data":"f077f901e20ef977b24913c45e0cbe81788e7945c8ce133cc5f49c64a479eedf"} Jul 06 19:21:27.047174 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.047074 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerStarted","Data":"d6a8deb5ba9fc48d8a15d0e1a0a5240c5e2f1c3c4ea99070cc59082cfe99ee11"} Jul 06 19:21:27.047174 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.047116 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerStarted","Data":"9faca7f59a8d204663a6483d86bd1b53465af278bb611987ba974fd6bb514b11"} Jul 06 19:21:27.047174 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.047156 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerStarted","Data":"4c0bbbd52d0ff41276a9aefb7701279712afb13227118fcfbec08b1138670ed7"} Jul 06 19:21:27.048536 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.048510 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8rchd" event={"ID":"44d4a5a8-5d12-483d-939c-5e40d5d3658b","Type":"ContainerStarted","Data":"1b246fe4f1e99b0d4e7cf9e084a98af1d10f1de366b8586d1e0535522043713e"} Jul 06 19:21:27.050032 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.050013 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6c87777-zp7wc" event={"ID":"c32d39d7-ceae-487d-816a-65f8d80d2478","Type":"ContainerStarted","Data":"f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913"} Jul 06 19:21:27.050115 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.050044 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6c87777-zp7wc" event={"ID":"c32d39d7-ceae-487d-816a-65f8d80d2478","Type":"ContainerStarted","Data":"3ad05cf5235f77d4416eecd070cec71013a4409bf186c0f272c60da986a0f728"} Jul 06 19:21:27.051456 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.051434 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" event={"ID":"08c932a1-c259-48fd-9f3b-65c1c51a536f","Type":"ContainerStarted","Data":"dd2be2e711cd63508eb4f93ca13d7069505c771867498d68ce42f54f0d58f3b1"} Jul 06 19:21:27.051596 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.051578 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:27.053474 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.053121 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4s4w" event={"ID":"8d5c45b9-a5f0-421d-bf44-59461dfd8fd0","Type":"ContainerStarted","Data":"8a8e20647dfee07aa2af36d779817f1d711bc7d800a2f709c00105a568648d11"} Jul 06 19:21:27.053474 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.053179 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:21:27.056255 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.056235 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" event={"ID":"a1e27927-a3fb-4e73-854c-0175744772a6","Type":"ContainerStarted","Data":"e198857ef19f6a17c6ac6466412305a54df286d188ccf3e04b9d41edee01de51"} Jul 06 19:21:27.056373 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.056321 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" event={"ID":"a1e27927-a3fb-4e73-854c-0175744772a6","Type":"ContainerStarted","Data":"3a9162d400501f7624912ff267d5a037b4cd9079c9cc213e3a15926ef5d6a845"} Jul 06 19:21:27.056373 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.056363 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:27.056480 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.056402 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" event={"ID":"a1e27927-a3fb-4e73-854c-0175744772a6","Type":"ContainerStarted","Data":"afaa5ac8bc4dff85b20a794084f1f4c9372f2983ba2eb17cf480bbd5ada63450"} Jul 06 19:21:27.057066 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.057049 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" Jul 06 19:21:27.058064 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.058034 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerID="fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e" exitCode=0 Jul 06 19:21:27.058150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.058072 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerDied","Data":"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e"} Jul 06 19:21:27.062377 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.062336 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" Jul 06 19:21:27.062516 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.062503 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:27.063375 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.063340 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" podStartSLOduration=3.09478505 podStartE2EDuration="8.063327419s" totalImagesPullingTime="4.968542369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:27.062339029 +0000 UTC m=+74.122685956" watchObservedRunningTime="2026-07-06 19:21:27.063327419 +0000 UTC m=+74.123674344" Jul 06 19:21:27.093471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.093428 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54f87764f-gc958" podStartSLOduration=2.541250941 podStartE2EDuration="10.093414738s" totalImagesPullingTime="7.552163797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:27.092745261 +0000 UTC m=+74.153092204" watchObservedRunningTime="2026-07-06 19:21:27.093414738 +0000 UTC m=+74.153761664" Jul 06 19:21:27.126807 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.126759 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.0859547 podStartE2EDuration="11.126738157s" totalImagesPullingTime="8.020657398s" totalInitContainerRuntime="20.126059ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:27.12539123 +0000 UTC m=+74.185738158" watchObservedRunningTime="2026-07-06 19:21:27.126738157 +0000 UTC m=+74.187085084" Jul 06 19:21:27.169975 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.169888 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h4s4w" podStartSLOduration=67.455270135 podStartE2EDuration="1m14.169870964s" totalImagesPullingTime="6.714600829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:27.1684762 +0000 UTC m=+74.228823129" watchObservedRunningTime="2026-07-06 19:21:27.169870964 +0000 UTC m=+74.230217889" Jul 06 19:21:27.185574 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.185526 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cd6c87777-zp7wc" podStartSLOduration=4.185512804 podStartE2EDuration="4.185512804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:27.184540192 +0000 UTC m=+74.244887133" watchObservedRunningTime="2026-07-06 19:21:27.185512804 +0000 UTC m=+74.245859710" Jul 06 19:21:27.201421 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.201381 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cswwx" podStartSLOduration=2.345882618 podStartE2EDuration="7.201362835s" totalImagesPullingTime="4.855480217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:27.200119508 +0000 UTC m=+74.260466434" watchObservedRunningTime="2026-07-06 19:21:27.201362835 +0000 UTC m=+74.261709762" Jul 06 19:21:27.220250 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:27.220214 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8rchd" podStartSLOduration=67.474732078 podStartE2EDuration="1m14.220201128s" totalImagesPullingTime="6.74546905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:27.218346671 +0000 UTC m=+74.278693599" watchObservedRunningTime="2026-07-06 19:21:27.220201128 +0000 UTC m=+74.280548077" Jul 06 19:21:31.089071 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:31.089037 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerStarted","Data":"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7"} Jul 06 19:21:31.089530 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:31.089156 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerStarted","Data":"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061"} Jul 06 19:21:32.095835 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:32.095798 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerStarted","Data":"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3"} Jul 06 19:21:32.096176 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:32.095877 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerStarted","Data":"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875"} Jul 06 19:21:32.096176 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:32.095918 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerStarted","Data":"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae"} Jul 06 19:21:32.096176 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:32.095957 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerStarted","Data":"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6"} Jul 06 19:21:32.130763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:32.130700 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.266227755 podStartE2EDuration="11.130678998s" totalImagesPullingTime="3.807077085s" totalInitContainerRuntime="57.374158ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:32.127130257 +0000 UTC m=+79.187477207" watchObservedRunningTime="2026-07-06 19:21:32.130678998 +0000 UTC m=+79.191025930" Jul 06 19:21:32.343719 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:32.343688 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:33.728386 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:33.728249 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:33.728386 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:33.728308 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:33.732750 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:33.732729 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:34.107508 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:34.107484 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:21:34.181437 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:34.181409 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d4f78764c-9xm54"] Jul 06 19:21:40.273368 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:40.273333 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:40.273768 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:40.273394 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:21:48.395176 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.395113 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7d5b6bff5-zzwm6" podUID="e7909354-3757-455a-9314-c734822f4fd0" containerName="console" containerID="cri-o://21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41" gracePeriod=15 Jul 06 19:21:48.631560 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.631538 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:48.716365 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716293 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-serving-cert\") pod \"e7909354-3757-455a-9314-c734822f4fd0\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " Jul 06 19:21:48.716365 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716326 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-oauth-serving-cert\") pod \"e7909354-3757-455a-9314-c734822f4fd0\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " Jul 06 19:21:48.716365 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716351 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-service-ca\") pod \"e7909354-3757-455a-9314-c734822f4fd0\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " Jul 06 19:21:48.716610 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716371 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-console-config\") pod \"e7909354-3757-455a-9314-c734822f4fd0\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " Jul 06 19:21:48.716610 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716507 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-oauth-config\") pod \"e7909354-3757-455a-9314-c734822f4fd0\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " Jul 06 19:21:48.716610 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716561 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m2hr\" (UniqueName: \"kubernetes.io/projected/e7909354-3757-455a-9314-c734822f4fd0-kube-api-access-2m2hr\") pod \"e7909354-3757-455a-9314-c734822f4fd0\" (UID: \"e7909354-3757-455a-9314-c734822f4fd0\") " Jul 06 19:21:48.716805 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716778 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-service-ca" (OuterVolumeSpecName: "service-ca") pod "e7909354-3757-455a-9314-c734822f4fd0" (UID: "e7909354-3757-455a-9314-c734822f4fd0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:21:48.716805 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716791 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-console-config" (OuterVolumeSpecName: "console-config") pod "e7909354-3757-455a-9314-c734822f4fd0" (UID: "e7909354-3757-455a-9314-c734822f4fd0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:21:48.716904 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716784 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e7909354-3757-455a-9314-c734822f4fd0" (UID: "e7909354-3757-455a-9314-c734822f4fd0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:21:48.716904 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716884 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-service-ca\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:48.716985 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716904 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-console-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:48.716985 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.716915 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7909354-3757-455a-9314-c734822f4fd0-oauth-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:48.718874 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.718845 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7909354-3757-455a-9314-c734822f4fd0-kube-api-access-2m2hr" (OuterVolumeSpecName: "kube-api-access-2m2hr") pod "e7909354-3757-455a-9314-c734822f4fd0" (UID: "e7909354-3757-455a-9314-c734822f4fd0"). InnerVolumeSpecName "kube-api-access-2m2hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:21:48.718874 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.718855 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e7909354-3757-455a-9314-c734822f4fd0" (UID: "e7909354-3757-455a-9314-c734822f4fd0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:21:48.719014 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.718918 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e7909354-3757-455a-9314-c734822f4fd0" (UID: "e7909354-3757-455a-9314-c734822f4fd0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:21:48.817575 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.817543 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:48.817575 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.817567 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7909354-3757-455a-9314-c734822f4fd0-console-oauth-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:48.817575 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:48.817577 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2hr\" (UniqueName: \"kubernetes.io/projected/e7909354-3757-455a-9314-c734822f4fd0-kube-api-access-2m2hr\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:49.165566 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.165529 2571 generic.go:356] "Generic (PLEG): container finished" podID="e7909354-3757-455a-9314-c734822f4fd0" containerID="21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41" exitCode=0 Jul 06 19:21:49.165725 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.165585 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5b6bff5-zzwm6" Jul 06 19:21:49.165725 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.165608 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5b6bff5-zzwm6" event={"ID":"e7909354-3757-455a-9314-c734822f4fd0","Type":"ContainerDied","Data":"21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41"} Jul 06 19:21:49.165725 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.165679 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5b6bff5-zzwm6" event={"ID":"e7909354-3757-455a-9314-c734822f4fd0","Type":"ContainerDied","Data":"0a3ac444e0bbdab0893e8ad58b798871fab0fb1c45c445e3d2ea888abe27361c"} Jul 06 19:21:49.165725 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.165710 2571 scope.go:117] "RemoveContainer" containerID="21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41" Jul 06 19:21:49.175485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.175468 2571 scope.go:117] "RemoveContainer" containerID="21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41" Jul 06 19:21:49.175727 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:21:49.175707 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41\": container with ID starting with 21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41 not found: ID does not exist" containerID="21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41" Jul 06 19:21:49.175772 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.175735 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41"} err="failed to get container status \"21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41\": rpc error: code = NotFound desc = could not find container \"21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41\": container with ID starting with 21507428a9d4323508fc866e500b3a7dd7e289aed0938fe5ad500eb8edb79b41 not found: ID does not exist" Jul 06 19:21:49.186157 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.186137 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d5b6bff5-zzwm6"] Jul 06 19:21:49.190795 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.190771 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d5b6bff5-zzwm6"] Jul 06 19:21:49.460028 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:49.459956 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7909354-3757-455a-9314-c734822f4fd0" path="/var/lib/kubelet/pods/e7909354-3757-455a-9314-c734822f4fd0/volumes" Jul 06 19:21:58.064760 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:58.064730 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h4s4w" Jul 06 19:21:59.207533 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.207467 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6d4f78764c-9xm54" podUID="5666d89c-e698-4a92-8c1f-401596cbda92" containerName="console" containerID="cri-o://e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d" gracePeriod=15 Jul 06 19:21:59.489529 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.489466 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:21:59.606950 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.606919 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-serving-cert\") pod \"5666d89c-e698-4a92-8c1f-401596cbda92\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " Jul 06 19:21:59.607103 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.606972 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-oauth-config\") pod \"5666d89c-e698-4a92-8c1f-401596cbda92\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " Jul 06 19:21:59.607103 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.607003 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-oauth-serving-cert\") pod \"5666d89c-e698-4a92-8c1f-401596cbda92\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " Jul 06 19:21:59.607103 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.607035 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snsdq\" (UniqueName: \"kubernetes.io/projected/5666d89c-e698-4a92-8c1f-401596cbda92-kube-api-access-snsdq\") pod \"5666d89c-e698-4a92-8c1f-401596cbda92\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " Jul 06 19:21:59.607103 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.607074 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-trusted-ca-bundle\") pod \"5666d89c-e698-4a92-8c1f-401596cbda92\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " Jul 06 19:21:59.607103 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.607100 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-console-config\") pod \"5666d89c-e698-4a92-8c1f-401596cbda92\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " Jul 06 19:21:59.607384 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.607202 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-service-ca\") pod \"5666d89c-e698-4a92-8c1f-401596cbda92\" (UID: \"5666d89c-e698-4a92-8c1f-401596cbda92\") " Jul 06 19:21:59.607562 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.607519 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5666d89c-e698-4a92-8c1f-401596cbda92" (UID: "5666d89c-e698-4a92-8c1f-401596cbda92"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:21:59.607562 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.607538 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5666d89c-e698-4a92-8c1f-401596cbda92" (UID: "5666d89c-e698-4a92-8c1f-401596cbda92"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:21:59.607748 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.607675 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-console-config" (OuterVolumeSpecName: "console-config") pod "5666d89c-e698-4a92-8c1f-401596cbda92" (UID: "5666d89c-e698-4a92-8c1f-401596cbda92"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:21:59.607808 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.607781 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-service-ca" (OuterVolumeSpecName: "service-ca") pod "5666d89c-e698-4a92-8c1f-401596cbda92" (UID: "5666d89c-e698-4a92-8c1f-401596cbda92"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:21:59.609491 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.609470 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5666d89c-e698-4a92-8c1f-401596cbda92" (UID: "5666d89c-e698-4a92-8c1f-401596cbda92"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:21:59.609870 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.609846 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5666d89c-e698-4a92-8c1f-401596cbda92-kube-api-access-snsdq" (OuterVolumeSpecName: "kube-api-access-snsdq") pod "5666d89c-e698-4a92-8c1f-401596cbda92" (UID: "5666d89c-e698-4a92-8c1f-401596cbda92"). InnerVolumeSpecName "kube-api-access-snsdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:21:59.609870 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.609856 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5666d89c-e698-4a92-8c1f-401596cbda92" (UID: "5666d89c-e698-4a92-8c1f-401596cbda92"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:21:59.708589 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.708566 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-service-ca\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:59.708589 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.708587 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:59.708732 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.708597 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5666d89c-e698-4a92-8c1f-401596cbda92-console-oauth-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:59.708732 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.708608 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-oauth-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:59.708732 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.708621 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snsdq\" (UniqueName: \"kubernetes.io/projected/5666d89c-e698-4a92-8c1f-401596cbda92-kube-api-access-snsdq\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:59.708732 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.708631 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-trusted-ca-bundle\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:21:59.708732 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:21:59.708639 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5666d89c-e698-4a92-8c1f-401596cbda92-console-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:00.210551 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.210514 2571 generic.go:356] "Generic (PLEG): container finished" podID="5666d89c-e698-4a92-8c1f-401596cbda92" containerID="e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d" exitCode=0 Jul 06 19:22:00.210958 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.210556 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4f78764c-9xm54" event={"ID":"5666d89c-e698-4a92-8c1f-401596cbda92","Type":"ContainerDied","Data":"e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d"} Jul 06 19:22:00.210958 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.210564 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4f78764c-9xm54" Jul 06 19:22:00.210958 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.210604 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4f78764c-9xm54" event={"ID":"5666d89c-e698-4a92-8c1f-401596cbda92","Type":"ContainerDied","Data":"29e27635ca7961aa71b63a507ed2df6be149635b81bc234ed68f0f05ba0ef894"} Jul 06 19:22:00.210958 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.210631 2571 scope.go:117] "RemoveContainer" containerID="e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d" Jul 06 19:22:00.219571 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.219553 2571 scope.go:117] "RemoveContainer" containerID="e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d" Jul 06 19:22:00.219827 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:22:00.219796 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d\": container with ID starting with e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d not found: ID does not exist" containerID="e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d" Jul 06 19:22:00.219892 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.219833 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d"} err="failed to get container status \"e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d\": rpc error: code = NotFound desc = could not find container \"e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d\": container with ID starting with e86f09dd9811356038f927b1d7b819955b6dfc76e017931723eab486e91d669d not found: ID does not exist" Jul 06 19:22:00.232519 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.232498 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d4f78764c-9xm54"] Jul 06 19:22:00.236456 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.236433 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d4f78764c-9xm54"] Jul 06 19:22:00.278456 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.278433 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:22:00.282066 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:00.282046 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bf5f9c4b9-kvwtp" Jul 06 19:22:01.463415 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:01.461396 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5666d89c-e698-4a92-8c1f-401596cbda92" path="/var/lib/kubelet/pods/5666d89c-e698-4a92-8c1f-401596cbda92/volumes" Jul 06 19:22:22.344117 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:22.344074 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:22.363707 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:22.363684 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:23.321107 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:23.321069 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:36.159528 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.159487 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:22:36.160050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.159779 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="alertmanager" containerID="cri-o://4c0bbbd52d0ff41276a9aefb7701279712afb13227118fcfbec08b1138670ed7" gracePeriod=120 Jul 06 19:22:36.160050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.159835 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="prom-label-proxy" containerID="cri-o://1ed08eb16375ef3f950c0d2887f598e232086c1ee2fdfa127aeb19519a03da14" gracePeriod=120 Jul 06 19:22:36.160050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.159835 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy" containerID="cri-o://f077f901e20ef977b24913c45e0cbe81788e7945c8ce133cc5f49c64a479eedf" gracePeriod=120 Jul 06 19:22:36.160050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.159851 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy-metric" containerID="cri-o://0eee0b12901158bd6c7b9ed5eba00dcb50667fb9bb554287a8a281d0139c2619" gracePeriod=120 Jul 06 19:22:36.160050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.159886 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="config-reloader" containerID="cri-o://9faca7f59a8d204663a6483d86bd1b53465af278bb611987ba974fd6bb514b11" gracePeriod=120 Jul 06 19:22:36.160050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.159849 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy-web" containerID="cri-o://d6a8deb5ba9fc48d8a15d0e1a0a5240c5e2f1c3c4ea99070cc59082cfe99ee11" gracePeriod=120 Jul 06 19:22:36.359457 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359422 2571 generic.go:356] "Generic (PLEG): container finished" podID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerID="1ed08eb16375ef3f950c0d2887f598e232086c1ee2fdfa127aeb19519a03da14" exitCode=0 Jul 06 19:22:36.359457 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359445 2571 generic.go:356] "Generic (PLEG): container finished" podID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerID="0eee0b12901158bd6c7b9ed5eba00dcb50667fb9bb554287a8a281d0139c2619" exitCode=0 Jul 06 19:22:36.359457 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359452 2571 generic.go:356] "Generic (PLEG): container finished" podID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerID="f077f901e20ef977b24913c45e0cbe81788e7945c8ce133cc5f49c64a479eedf" exitCode=0 Jul 06 19:22:36.359457 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359458 2571 generic.go:356] "Generic (PLEG): container finished" podID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerID="9faca7f59a8d204663a6483d86bd1b53465af278bb611987ba974fd6bb514b11" exitCode=0 Jul 06 19:22:36.359457 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359463 2571 generic.go:356] "Generic (PLEG): container finished" podID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerID="4c0bbbd52d0ff41276a9aefb7701279712afb13227118fcfbec08b1138670ed7" exitCode=0 Jul 06 19:22:36.359784 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359492 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerDied","Data":"1ed08eb16375ef3f950c0d2887f598e232086c1ee2fdfa127aeb19519a03da14"} Jul 06 19:22:36.359784 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359551 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerDied","Data":"0eee0b12901158bd6c7b9ed5eba00dcb50667fb9bb554287a8a281d0139c2619"} Jul 06 19:22:36.359784 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359577 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerDied","Data":"f077f901e20ef977b24913c45e0cbe81788e7945c8ce133cc5f49c64a479eedf"} Jul 06 19:22:36.359784 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359603 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerDied","Data":"9faca7f59a8d204663a6483d86bd1b53465af278bb611987ba974fd6bb514b11"} Jul 06 19:22:36.359784 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:36.359626 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerDied","Data":"4c0bbbd52d0ff41276a9aefb7701279712afb13227118fcfbec08b1138670ed7"} Jul 06 19:22:37.366806 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.366775 2571 generic.go:356] "Generic (PLEG): container finished" podID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerID="d6a8deb5ba9fc48d8a15d0e1a0a5240c5e2f1c3c4ea99070cc59082cfe99ee11" exitCode=0 Jul 06 19:22:37.366806 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.366810 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerDied","Data":"d6a8deb5ba9fc48d8a15d0e1a0a5240c5e2f1c3c4ea99070cc59082cfe99ee11"} Jul 06 19:22:37.403678 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.403657 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:37.544954 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.544925 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545123 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.544966 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-volume\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545123 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545025 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-main-tls\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545123 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545061 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545123 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545092 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-main-db\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545369 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545225 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-tls-assets\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545369 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545308 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-trusted-ca-bundle\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545369 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545364 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-cluster-tls-config\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545591 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545398 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545591 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545429 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-metrics-client-ca\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545591 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545465 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-out\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545591 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545471 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:22:37.545591 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545500 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqzrs\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-kube-api-access-vqzrs\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.545591 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545531 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-web-config\") pod \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\" (UID: \"9341c82a-df20-483b-ad3d-dc16ad3e4b49\") " Jul 06 19:22:37.546184 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.545937 2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-main-db\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.546549 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.546381 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:22:37.547504 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.547476 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:22:37.548303 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.548246 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:37.548575 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.548551 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:37.548835 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.548811 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:37.549149 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.549126 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-kube-api-access-vqzrs" (OuterVolumeSpecName: "kube-api-access-vqzrs") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "kube-api-access-vqzrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:22:37.549687 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.549639 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:37.549861 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.549840 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-volume" (OuterVolumeSpecName: "config-volume") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:37.550436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.550412 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-out" (OuterVolumeSpecName: "config-out") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:22:37.550523 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.550439 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:22:37.554016 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.553920 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:37.561462 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.561348 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-web-config" (OuterVolumeSpecName: "web-config") pod "9341c82a-df20-483b-ad3d-dc16ad3e4b49" (UID: "9341c82a-df20-483b-ad3d-dc16ad3e4b49"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:37.646959 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.646925 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.646959 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.646955 2571 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-volume\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.646959 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.646965 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-main-tls\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.647169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.646974 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.647169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.646983 2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-tls-assets\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.647169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.646992 2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.647169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.647000 2571 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-cluster-tls-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.647169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.647010 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.647169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.647018 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9341c82a-df20-483b-ad3d-dc16ad3e4b49-metrics-client-ca\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.647169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.647026 2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9341c82a-df20-483b-ad3d-dc16ad3e4b49-config-out\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.647169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.647034 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqzrs\" (UniqueName: \"kubernetes.io/projected/9341c82a-df20-483b-ad3d-dc16ad3e4b49-kube-api-access-vqzrs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:37.647169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:37.647042 2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9341c82a-df20-483b-ad3d-dc16ad3e4b49-web-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:38.373329 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.373296 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9341c82a-df20-483b-ad3d-dc16ad3e4b49","Type":"ContainerDied","Data":"d781374d479a769808a082508592ad39292ab6624b037b40fa984f6e737b19a4"} Jul 06 19:22:38.373329 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.373326 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.373811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.373428 2571 scope.go:117] "RemoveContainer" containerID="1ed08eb16375ef3f950c0d2887f598e232086c1ee2fdfa127aeb19519a03da14" Jul 06 19:22:38.382031 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.382004 2571 scope.go:117] "RemoveContainer" containerID="0eee0b12901158bd6c7b9ed5eba00dcb50667fb9bb554287a8a281d0139c2619" Jul 06 19:22:38.389079 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.389060 2571 scope.go:117] "RemoveContainer" containerID="f077f901e20ef977b24913c45e0cbe81788e7945c8ce133cc5f49c64a479eedf" Jul 06 19:22:38.396028 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.396009 2571 scope.go:117] "RemoveContainer" containerID="d6a8deb5ba9fc48d8a15d0e1a0a5240c5e2f1c3c4ea99070cc59082cfe99ee11" Jul 06 19:22:38.399156 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.399122 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:22:38.403592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.403535 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:22:38.404594 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.404565 2571 scope.go:117] "RemoveContainer" containerID="9faca7f59a8d204663a6483d86bd1b53465af278bb611987ba974fd6bb514b11" Jul 06 19:22:38.411665 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.411648 2571 scope.go:117] "RemoveContainer" containerID="4c0bbbd52d0ff41276a9aefb7701279712afb13227118fcfbec08b1138670ed7" Jul 06 19:22:38.418595 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.418576 2571 scope.go:117] "RemoveContainer" containerID="8f7ced3a04b6caf97e5d30c642b38af80e04c3ebc1a3ece2fc9f2567286455ac" Jul 06 19:22:38.433880 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.433860 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:22:38.434166 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434154 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="config-reloader" Jul 06 19:22:38.434206 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434167 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="config-reloader" Jul 06 19:22:38.434206 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434180 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5666d89c-e698-4a92-8c1f-401596cbda92" containerName="console" Jul 06 19:22:38.434206 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434186 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5666d89c-e698-4a92-8c1f-401596cbda92" containerName="console" Jul 06 19:22:38.434206 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434193 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy" Jul 06 19:22:38.434206 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434198 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy" Jul 06 19:22:38.434206 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434206 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy-metric" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434212 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy-metric" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434218 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7909354-3757-455a-9314-c734822f4fd0" containerName="console" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434222 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7909354-3757-455a-9314-c734822f4fd0" containerName="console" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434230 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="init-config-reloader" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434235 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="init-config-reloader" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434241 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy-web" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434245 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy-web" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434254 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="alertmanager" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434259 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="alertmanager" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434265 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="prom-label-proxy" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434285 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="prom-label-proxy" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434334 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="alertmanager" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434343 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5666d89c-e698-4a92-8c1f-401596cbda92" containerName="console" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434350 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434355 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7909354-3757-455a-9314-c734822f4fd0" containerName="console" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434361 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy-web" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434366 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="prom-label-proxy" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434370 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="config-reloader" Jul 06 19:22:38.434397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.434376 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" containerName="kube-rbac-proxy-metric" Jul 06 19:22:38.437791 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.437776 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.440374 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.440344 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 19:22:38.440374 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.440367 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 19:22:38.440518 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.440408 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 19:22:38.440697 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.440673 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 19:22:38.441076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.441058 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 19:22:38.441179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.441149 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 19:22:38.441250 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.441202 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 19:22:38.441250 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.441210 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s67qf\"" Jul 06 19:22:38.441250 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.441150 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 19:22:38.447716 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.447698 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 19:22:38.453995 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.453972 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:22:38.555674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.555674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555619 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a4f69b-68d3-475f-ab99-f38386c1383b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.555674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.555674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555676 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a4f69b-68d3-475f-ab99-f38386c1383b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.556011 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555783 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-config-volume\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.556011 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555820 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.556011 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555854 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9nd9\" (UniqueName: \"kubernetes.io/projected/06a4f69b-68d3-475f-ab99-f38386c1383b-kube-api-access-x9nd9\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.556011 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555890 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a4f69b-68d3-475f-ab99-f38386c1383b-config-out\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.556011 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555916 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.556011 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555958 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a4f69b-68d3-475f-ab99-f38386c1383b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.556011 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.555992 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.556255 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.556029 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-web-config\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.556255 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.556054 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06a4f69b-68d3-475f-ab99-f38386c1383b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.656610 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656573 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9nd9\" (UniqueName: \"kubernetes.io/projected/06a4f69b-68d3-475f-ab99-f38386c1383b-kube-api-access-x9nd9\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.656779 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a4f69b-68d3-475f-ab99-f38386c1383b-config-out\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.656779 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656636 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.656779 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a4f69b-68d3-475f-ab99-f38386c1383b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.656779 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656686 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.656779 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656710 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-web-config\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.656779 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656726 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06a4f69b-68d3-475f-ab99-f38386c1383b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.656779 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656747 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.656779 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656770 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a4f69b-68d3-475f-ab99-f38386c1383b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.657228 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656823 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.657228 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a4f69b-68d3-475f-ab99-f38386c1383b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.657228 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.656960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-config-volume\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.657228 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.657006 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.658122 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.657569 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a4f69b-68d3-475f-ab99-f38386c1383b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.658122 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.657636 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a4f69b-68d3-475f-ab99-f38386c1383b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.658122 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.657811 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06a4f69b-68d3-475f-ab99-f38386c1383b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.659958 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.659936 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.660102 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.660078 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.660175 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.660122 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.660479 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.660456 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.660609 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.660588 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a4f69b-68d3-475f-ab99-f38386c1383b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.660609 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.660603 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a4f69b-68d3-475f-ab99-f38386c1383b-config-out\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.660715 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.660696 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-web-config\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.660928 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.660913 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.661991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.661970 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06a4f69b-68d3-475f-ab99-f38386c1383b-config-volume\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.664486 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.664467 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9nd9\" (UniqueName: \"kubernetes.io/projected/06a4f69b-68d3-475f-ab99-f38386c1383b-kube-api-access-x9nd9\") pod \"alertmanager-main-0\" (UID: \"06a4f69b-68d3-475f-ab99-f38386c1383b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.749707 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.749671 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:22:38.885500 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:38.885468 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:22:38.887341 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:22:38.887314 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a4f69b_68d3_475f_ab99_f38386c1383b.slice/crio-e50e16c3ba172dca6dd3abf5e3cfaed72e633554769aca672f6ac9eea8bdaf9f WatchSource:0}: Error finding container e50e16c3ba172dca6dd3abf5e3cfaed72e633554769aca672f6ac9eea8bdaf9f: Status 404 returned error can't find the container with id e50e16c3ba172dca6dd3abf5e3cfaed72e633554769aca672f6ac9eea8bdaf9f Jul 06 19:22:39.378644 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:39.378611 2571 generic.go:356] "Generic (PLEG): container finished" podID="06a4f69b-68d3-475f-ab99-f38386c1383b" containerID="ebba24a4396a8a3c6052ad1839c7b9316ebfdcd412ffc4f69711148dac94d8ef" exitCode=0 Jul 06 19:22:39.379062 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:39.378701 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06a4f69b-68d3-475f-ab99-f38386c1383b","Type":"ContainerDied","Data":"ebba24a4396a8a3c6052ad1839c7b9316ebfdcd412ffc4f69711148dac94d8ef"} Jul 06 19:22:39.379062 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:39.378772 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06a4f69b-68d3-475f-ab99-f38386c1383b","Type":"ContainerStarted","Data":"e50e16c3ba172dca6dd3abf5e3cfaed72e633554769aca672f6ac9eea8bdaf9f"} Jul 06 19:22:39.461519 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:39.461496 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9341c82a-df20-483b-ad3d-dc16ad3e4b49" path="/var/lib/kubelet/pods/9341c82a-df20-483b-ad3d-dc16ad3e4b49/volumes" Jul 06 19:22:40.100573 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.100532 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-64bd95bbdb-gs77h"] Jul 06 19:22:40.103655 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.103629 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.106959 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.106927 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 19:22:40.107105 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.107087 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 19:22:40.107186 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.107167 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lk7f7\"" Jul 06 19:22:40.107239 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.107196 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 19:22:40.107239 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.107087 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 19:22:40.107239 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.107227 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 19:22:40.119311 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.118932 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 19:22:40.123447 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.123409 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64bd95bbdb-gs77h"] Jul 06 19:22:40.170674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.170640 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/088fd4a9-6953-4809-bc48-d98f20202fe8-serving-certs-ca-bundle\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.170823 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.170687 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-federate-client-tls\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.170823 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.170706 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-telemeter-client-tls\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.170823 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.170762 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-secret-telemeter-client\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.170823 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.170799 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/088fd4a9-6953-4809-bc48-d98f20202fe8-metrics-client-ca\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.170823 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.170824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26sdw\" (UniqueName: \"kubernetes.io/projected/088fd4a9-6953-4809-bc48-d98f20202fe8-kube-api-access-26sdw\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.170993 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.170855 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.170993 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.170920 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/088fd4a9-6953-4809-bc48-d98f20202fe8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.272404 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.272373 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/088fd4a9-6953-4809-bc48-d98f20202fe8-serving-certs-ca-bundle\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.272578 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.272422 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-federate-client-tls\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.272578 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.272454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-telemeter-client-tls\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.272578 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.272521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-secret-telemeter-client\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.272578 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.272570 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/088fd4a9-6953-4809-bc48-d98f20202fe8-metrics-client-ca\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.272763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.272610 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26sdw\" (UniqueName: \"kubernetes.io/projected/088fd4a9-6953-4809-bc48-d98f20202fe8-kube-api-access-26sdw\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.272763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.272670 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.272763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.272716 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/088fd4a9-6953-4809-bc48-d98f20202fe8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.273368 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.273322 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/088fd4a9-6953-4809-bc48-d98f20202fe8-serving-certs-ca-bundle\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.274003 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.273978 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/088fd4a9-6953-4809-bc48-d98f20202fe8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.274320 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.274231 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/088fd4a9-6953-4809-bc48-d98f20202fe8-metrics-client-ca\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.275781 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.275741 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-federate-client-tls\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.276305 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.276252 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-secret-telemeter-client\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.276397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.276369 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.276477 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.276458 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/088fd4a9-6953-4809-bc48-d98f20202fe8-telemeter-client-tls\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.283165 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.283145 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26sdw\" (UniqueName: \"kubernetes.io/projected/088fd4a9-6953-4809-bc48-d98f20202fe8-kube-api-access-26sdw\") pod \"telemeter-client-64bd95bbdb-gs77h\" (UID: \"088fd4a9-6953-4809-bc48-d98f20202fe8\") " pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.386620 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.386533 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06a4f69b-68d3-475f-ab99-f38386c1383b","Type":"ContainerStarted","Data":"c837bf412b35ca2fc04fddcfde9704b0893d5a9446a5d2ddb2e9121ac7741b1f"} Jul 06 19:22:40.386620 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.386598 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06a4f69b-68d3-475f-ab99-f38386c1383b","Type":"ContainerStarted","Data":"81ddd77b653ba60685005423b249b38ce5b3be1a379dc39fcea7e9796e0057e3"} Jul 06 19:22:40.387017 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.386624 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06a4f69b-68d3-475f-ab99-f38386c1383b","Type":"ContainerStarted","Data":"d1ae4397fd6636fd37a074a23e9f97ce3dfe96efab83a2e6eb96e1126742c34e"} Jul 06 19:22:40.387017 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.386655 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06a4f69b-68d3-475f-ab99-f38386c1383b","Type":"ContainerStarted","Data":"de0899078b9438d5037275b2b2108f8f9c3f22333b5d74290b33f54c1740afb8"} Jul 06 19:22:40.387017 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.386679 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06a4f69b-68d3-475f-ab99-f38386c1383b","Type":"ContainerStarted","Data":"60cdfd3ed0c9c28a948fdff45c39d6b1c6244e06098437693eada0fed207026c"} Jul 06 19:22:40.387017 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.386703 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06a4f69b-68d3-475f-ab99-f38386c1383b","Type":"ContainerStarted","Data":"134163c5e90b8e8d713810d273afddccdbdc4cf231017b6b8d9b64f1b9078ad4"} Jul 06 19:22:40.419436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.419373 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.40364119 podStartE2EDuration="2.41935173s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.71054ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:40.417842465 +0000 UTC m=+147.478189391" watchObservedRunningTime="2026-07-06 19:22:40.41935173 +0000 UTC m=+147.479698664" Jul 06 19:22:40.423914 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.423882 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" Jul 06 19:22:40.430969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.427396 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:22:40.430969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.427803 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" gracePeriod=600 Jul 06 19:22:40.430969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.428063 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy-web" containerID="cri-o://ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" gracePeriod=600 Jul 06 19:22:40.430969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.428112 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="thanos-sidecar" containerID="cri-o://abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" gracePeriod=600 Jul 06 19:22:40.430969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.428169 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy" containerID="cri-o://e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" gracePeriod=600 Jul 06 19:22:40.430969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.428245 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="config-reloader" containerID="cri-o://2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" gracePeriod=600 Jul 06 19:22:40.430969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.428367 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="prometheus" containerID="cri-o://9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" gracePeriod=600 Jul 06 19:22:40.606630 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.606600 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64bd95bbdb-gs77h"] Jul 06 19:22:40.608420 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:22:40.608390 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088fd4a9_6953_4809_bc48_d98f20202fe8.slice/crio-e643964d887b3e0dc09907e06f08b53ef27d8322446a54356ebf1c46c444823d WatchSource:0}: Error finding container e643964d887b3e0dc09907e06f08b53ef27d8322446a54356ebf1c46c444823d: Status 404 returned error can't find the container with id e643964d887b3e0dc09907e06f08b53ef27d8322446a54356ebf1c46c444823d Jul 06 19:22:40.696888 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.696861 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:40.779593 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779559 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-web-config\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.779593 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779594 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-tls\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.779811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779628 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nq8q\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-kube-api-access-7nq8q\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.779811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779650 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-1\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.779811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779675 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-thanos-prometheus-http-client-file\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.779811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779705 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-config\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.779811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779738 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-config-out\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.779811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779772 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.779811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779802 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-kube-rbac-proxy\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.780148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779845 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-trusted-ca-bundle\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.780148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779872 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-2\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.780148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779907 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-kubelet-serving-ca-bundle\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.780148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779936 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-0\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.780148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.779981 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-serving-certs-ca-bundle\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.780148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.780014 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-grpc-tls\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.780148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.780043 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-db\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.780148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.780080 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:22:40.780148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.780096 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-metrics-client-certs\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.780768 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.780259 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.782265 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.781500 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:22:40.782265 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.781559 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:22:40.782265 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.781951 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:22:40.782265 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.782095 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:22:40.782265 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.782216 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-metrics-client-ca\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.782265 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.782258 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-tls-assets\") pod \"a8c1f96f-e470-4e61-a477-263bd51a7840\" (UID: \"a8c1f96f-e470-4e61-a477-263bd51a7840\") " Jul 06 19:22:40.782658 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.782626 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.782658 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.782647 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.782750 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.782662 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.782750 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.782679 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.782750 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.782693 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.783595 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.783566 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:22:40.784635 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.784459 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:22:40.784635 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.784596 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:22:40.784967 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.784936 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-config" (OuterVolumeSpecName: "config") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:40.785581 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.785472 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:40.785972 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.785930 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-kube-api-access-7nq8q" (OuterVolumeSpecName: "kube-api-access-7nq8q") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "kube-api-access-7nq8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:22:40.787718 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.787687 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:40.787813 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.787687 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-config-out" (OuterVolumeSpecName: "config-out") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:22:40.787989 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.787954 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:40.788099 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.788077 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:40.788403 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.788375 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:40.788542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.788523 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:22:40.788584 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.788534 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:40.789156 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.789116 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:40.798108 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.798084 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-web-config" (OuterVolumeSpecName: "web-config") pod "a8c1f96f-e470-4e61-a477-263bd51a7840" (UID: "a8c1f96f-e470-4e61-a477-263bd51a7840"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:22:40.883310 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883250 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883310 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883301 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nq8q\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-kube-api-access-7nq8q\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883310 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883315 2571 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883325 2571 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883334 2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-config-out\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883343 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883351 2571 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-kube-rbac-proxy\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883360 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883369 2571 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-grpc-tls\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883377 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c1f96f-e470-4e61-a477-263bd51a7840-prometheus-k8s-db\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883385 2571 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-metrics-client-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883395 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883403 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c1f96f-e470-4e61-a477-263bd51a7840-configmap-metrics-client-ca\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883411 2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c1f96f-e470-4e61-a477-263bd51a7840-tls-assets\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:40.883542 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:40.883420 2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c1f96f-e470-4e61-a477-263bd51a7840-web-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:22:41.391755 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.391720 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" event={"ID":"088fd4a9-6953-4809-bc48-d98f20202fe8","Type":"ContainerStarted","Data":"e643964d887b3e0dc09907e06f08b53ef27d8322446a54356ebf1c46c444823d"} Jul 06 19:22:41.394636 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394610 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerID="7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" exitCode=0 Jul 06 19:22:41.394636 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394636 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerID="e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" exitCode=0 Jul 06 19:22:41.394786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394642 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerID="ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" exitCode=0 Jul 06 19:22:41.394786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394648 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerID="abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" exitCode=0 Jul 06 19:22:41.394786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394653 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerID="2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" exitCode=0 Jul 06 19:22:41.394786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394660 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerID="9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" exitCode=0 Jul 06 19:22:41.394786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394669 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerDied","Data":"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875"} Jul 06 19:22:41.394786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394685 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.394786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394747 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerDied","Data":"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae"} Jul 06 19:22:41.394786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394779 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerDied","Data":"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6"} Jul 06 19:22:41.395015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394814 2571 scope.go:117] "RemoveContainer" containerID="7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" Jul 06 19:22:41.395015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerDied","Data":"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3"} Jul 06 19:22:41.395015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394920 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerDied","Data":"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7"} Jul 06 19:22:41.395015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394953 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerDied","Data":"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061"} Jul 06 19:22:41.395015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.394990 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8c1f96f-e470-4e61-a477-263bd51a7840","Type":"ContainerDied","Data":"d9348de7d37c35fa84549fcf4983b46feb384f8798703b691f78f79f95bf326c"} Jul 06 19:22:41.403699 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.403679 2571 scope.go:117] "RemoveContainer" containerID="e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" Jul 06 19:22:41.412180 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.412165 2571 scope.go:117] "RemoveContainer" containerID="ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" Jul 06 19:22:41.418931 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.418910 2571 scope.go:117] "RemoveContainer" containerID="abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" Jul 06 19:22:41.424212 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.424192 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:22:41.426637 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.426624 2571 scope.go:117] "RemoveContainer" containerID="2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" Jul 06 19:22:41.430006 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.429986 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:22:41.433941 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.433791 2571 scope.go:117] "RemoveContainer" containerID="9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" Jul 06 19:22:41.440851 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.440834 2571 scope.go:117] "RemoveContainer" containerID="fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e" Jul 06 19:22:41.447568 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.447553 2571 scope.go:117] "RemoveContainer" containerID="7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" Jul 06 19:22:41.447825 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:22:41.447808 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": container with ID starting with 7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875 not found: ID does not exist" containerID="7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" Jul 06 19:22:41.447901 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.447830 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875"} err="failed to get container status \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": rpc error: code = NotFound desc = could not find container \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": container with ID starting with 7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875 not found: ID does not exist" Jul 06 19:22:41.447901 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.447849 2571 scope.go:117] "RemoveContainer" containerID="e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" Jul 06 19:22:41.448115 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:22:41.448098 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": container with ID starting with e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae not found: ID does not exist" containerID="e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" Jul 06 19:22:41.448179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.448128 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae"} err="failed to get container status \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": rpc error: code = NotFound desc = could not find container \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": container with ID starting with e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae not found: ID does not exist" Jul 06 19:22:41.448179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.448144 2571 scope.go:117] "RemoveContainer" containerID="ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" Jul 06 19:22:41.448431 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:22:41.448412 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": container with ID starting with ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6 not found: ID does not exist" containerID="ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" Jul 06 19:22:41.448502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.448435 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6"} err="failed to get container status \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": rpc error: code = NotFound desc = could not find container \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": container with ID starting with ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6 not found: ID does not exist" Jul 06 19:22:41.448502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.448449 2571 scope.go:117] "RemoveContainer" containerID="abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" Jul 06 19:22:41.448698 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:22:41.448679 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": container with ID starting with abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3 not found: ID does not exist" containerID="abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" Jul 06 19:22:41.448760 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.448705 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3"} err="failed to get container status \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": rpc error: code = NotFound desc = could not find container \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": container with ID starting with abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3 not found: ID does not exist" Jul 06 19:22:41.448760 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.448719 2571 scope.go:117] "RemoveContainer" containerID="2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" Jul 06 19:22:41.448987 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:22:41.448970 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": container with ID starting with 2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7 not found: ID does not exist" containerID="2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" Jul 06 19:22:41.449038 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.449001 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7"} err="failed to get container status \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": rpc error: code = NotFound desc = could not find container \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": container with ID starting with 2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7 not found: ID does not exist" Jul 06 19:22:41.449038 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.449023 2571 scope.go:117] "RemoveContainer" containerID="9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" Jul 06 19:22:41.449265 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:22:41.449249 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": container with ID starting with 9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061 not found: ID does not exist" containerID="9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" Jul 06 19:22:41.449338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.449292 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061"} err="failed to get container status \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": rpc error: code = NotFound desc = could not find container \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": container with ID starting with 9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061 not found: ID does not exist" Jul 06 19:22:41.449338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.449308 2571 scope.go:117] "RemoveContainer" containerID="fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e" Jul 06 19:22:41.449538 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:22:41.449520 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": container with ID starting with fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e not found: ID does not exist" containerID="fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e" Jul 06 19:22:41.449605 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.449545 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e"} err="failed to get container status \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": rpc error: code = NotFound desc = could not find container \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": container with ID starting with fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e not found: ID does not exist" Jul 06 19:22:41.449605 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.449567 2571 scope.go:117] "RemoveContainer" containerID="7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" Jul 06 19:22:41.449795 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.449779 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875"} err="failed to get container status \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": rpc error: code = NotFound desc = could not find container \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": container with ID starting with 7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875 not found: ID does not exist" Jul 06 19:22:41.449837 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.449796 2571 scope.go:117] "RemoveContainer" containerID="e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" Jul 06 19:22:41.449998 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.449980 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae"} err="failed to get container status \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": rpc error: code = NotFound desc = could not find container \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": container with ID starting with e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae not found: ID does not exist" Jul 06 19:22:41.450058 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.450000 2571 scope.go:117] "RemoveContainer" containerID="ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" Jul 06 19:22:41.450286 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.450248 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6"} err="failed to get container status \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": rpc error: code = NotFound desc = could not find container \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": container with ID starting with ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6 not found: ID does not exist" Jul 06 19:22:41.450358 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.450286 2571 scope.go:117] "RemoveContainer" containerID="abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" Jul 06 19:22:41.450760 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.450739 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3"} err="failed to get container status \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": rpc error: code = NotFound desc = could not find container \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": container with ID starting with abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3 not found: ID does not exist" Jul 06 19:22:41.450844 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.450768 2571 scope.go:117] "RemoveContainer" containerID="2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" Jul 06 19:22:41.451037 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.451016 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7"} err="failed to get container status \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": rpc error: code = NotFound desc = could not find container \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": container with ID starting with 2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7 not found: ID does not exist" Jul 06 19:22:41.451090 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.451038 2571 scope.go:117] "RemoveContainer" containerID="9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" Jul 06 19:22:41.451308 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.451264 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061"} err="failed to get container status \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": rpc error: code = NotFound desc = could not find container \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": container with ID starting with 9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061 not found: ID does not exist" Jul 06 19:22:41.451359 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.451310 2571 scope.go:117] "RemoveContainer" containerID="fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e" Jul 06 19:22:41.451522 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.451504 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e"} err="failed to get container status \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": rpc error: code = NotFound desc = could not find container \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": container with ID starting with fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e not found: ID does not exist" Jul 06 19:22:41.451591 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.451526 2571 scope.go:117] "RemoveContainer" containerID="7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" Jul 06 19:22:41.451758 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.451740 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875"} err="failed to get container status \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": rpc error: code = NotFound desc = could not find container \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": container with ID starting with 7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875 not found: ID does not exist" Jul 06 19:22:41.451825 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.451759 2571 scope.go:117] "RemoveContainer" containerID="e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" Jul 06 19:22:41.452003 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.451983 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae"} err="failed to get container status \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": rpc error: code = NotFound desc = could not find container \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": container with ID starting with e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae not found: ID does not exist" Jul 06 19:22:41.452054 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.452004 2571 scope.go:117] "RemoveContainer" containerID="ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" Jul 06 19:22:41.452219 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.452201 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6"} err="failed to get container status \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": rpc error: code = NotFound desc = could not find container \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": container with ID starting with ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6 not found: ID does not exist" Jul 06 19:22:41.452299 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.452220 2571 scope.go:117] "RemoveContainer" containerID="abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" Jul 06 19:22:41.452462 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.452446 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3"} err="failed to get container status \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": rpc error: code = NotFound desc = could not find container \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": container with ID starting with abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3 not found: ID does not exist" Jul 06 19:22:41.452533 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.452465 2571 scope.go:117] "RemoveContainer" containerID="2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" Jul 06 19:22:41.452666 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.452646 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7"} err="failed to get container status \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": rpc error: code = NotFound desc = could not find container \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": container with ID starting with 2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7 not found: ID does not exist" Jul 06 19:22:41.452716 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.452667 2571 scope.go:117] "RemoveContainer" containerID="9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" Jul 06 19:22:41.452904 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.452887 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061"} err="failed to get container status \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": rpc error: code = NotFound desc = could not find container \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": container with ID starting with 9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061 not found: ID does not exist" Jul 06 19:22:41.452946 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.452905 2571 scope.go:117] "RemoveContainer" containerID="fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e" Jul 06 19:22:41.453091 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.453073 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e"} err="failed to get container status \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": rpc error: code = NotFound desc = could not find container \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": container with ID starting with fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e not found: ID does not exist" Jul 06 19:22:41.453142 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.453091 2571 scope.go:117] "RemoveContainer" containerID="7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" Jul 06 19:22:41.453346 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.453327 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875"} err="failed to get container status \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": rpc error: code = NotFound desc = could not find container \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": container with ID starting with 7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875 not found: ID does not exist" Jul 06 19:22:41.453413 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.453348 2571 scope.go:117] "RemoveContainer" containerID="e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" Jul 06 19:22:41.453588 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.453572 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae"} err="failed to get container status \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": rpc error: code = NotFound desc = could not find container \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": container with ID starting with e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae not found: ID does not exist" Jul 06 19:22:41.453658 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.453589 2571 scope.go:117] "RemoveContainer" containerID="ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" Jul 06 19:22:41.453825 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.453807 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6"} err="failed to get container status \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": rpc error: code = NotFound desc = could not find container \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": container with ID starting with ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6 not found: ID does not exist" Jul 06 19:22:41.453874 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.453825 2571 scope.go:117] "RemoveContainer" containerID="abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" Jul 06 19:22:41.454036 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.454019 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3"} err="failed to get container status \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": rpc error: code = NotFound desc = could not find container \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": container with ID starting with abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3 not found: ID does not exist" Jul 06 19:22:41.454084 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.454037 2571 scope.go:117] "RemoveContainer" containerID="2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" Jul 06 19:22:41.454255 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.454237 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7"} err="failed to get container status \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": rpc error: code = NotFound desc = could not find container \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": container with ID starting with 2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7 not found: ID does not exist" Jul 06 19:22:41.454353 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.454256 2571 scope.go:117] "RemoveContainer" containerID="9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" Jul 06 19:22:41.454508 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.454489 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061"} err="failed to get container status \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": rpc error: code = NotFound desc = could not find container \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": container with ID starting with 9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061 not found: ID does not exist" Jul 06 19:22:41.454556 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.454509 2571 scope.go:117] "RemoveContainer" containerID="fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e" Jul 06 19:22:41.454763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.454746 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e"} err="failed to get container status \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": rpc error: code = NotFound desc = could not find container \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": container with ID starting with fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e not found: ID does not exist" Jul 06 19:22:41.454818 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.454763 2571 scope.go:117] "RemoveContainer" containerID="7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" Jul 06 19:22:41.455016 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455001 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875"} err="failed to get container status \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": rpc error: code = NotFound desc = could not find container \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": container with ID starting with 7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875 not found: ID does not exist" Jul 06 19:22:41.455016 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455016 2571 scope.go:117] "RemoveContainer" containerID="e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" Jul 06 19:22:41.455248 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455230 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae"} err="failed to get container status \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": rpc error: code = NotFound desc = could not find container \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": container with ID starting with e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae not found: ID does not exist" Jul 06 19:22:41.455312 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455249 2571 scope.go:117] "RemoveContainer" containerID="ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" Jul 06 19:22:41.455493 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455474 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6"} err="failed to get container status \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": rpc error: code = NotFound desc = could not find container \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": container with ID starting with ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6 not found: ID does not exist" Jul 06 19:22:41.455537 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455494 2571 scope.go:117] "RemoveContainer" containerID="abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" Jul 06 19:22:41.455719 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455702 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3"} err="failed to get container status \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": rpc error: code = NotFound desc = could not find container \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": container with ID starting with abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3 not found: ID does not exist" Jul 06 19:22:41.455779 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455720 2571 scope.go:117] "RemoveContainer" containerID="2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" Jul 06 19:22:41.455935 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455920 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7"} err="failed to get container status \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": rpc error: code = NotFound desc = could not find container \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": container with ID starting with 2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7 not found: ID does not exist" Jul 06 19:22:41.455977 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.455935 2571 scope.go:117] "RemoveContainer" containerID="9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" Jul 06 19:22:41.456178 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.456163 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061"} err="failed to get container status \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": rpc error: code = NotFound desc = could not find container \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": container with ID starting with 9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061 not found: ID does not exist" Jul 06 19:22:41.456305 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.456178 2571 scope.go:117] "RemoveContainer" containerID="fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e" Jul 06 19:22:41.456431 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.456415 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e"} err="failed to get container status \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": rpc error: code = NotFound desc = could not find container \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": container with ID starting with fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e not found: ID does not exist" Jul 06 19:22:41.456471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.456432 2571 scope.go:117] "RemoveContainer" containerID="7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875" Jul 06 19:22:41.456678 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.456663 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875"} err="failed to get container status \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": rpc error: code = NotFound desc = could not find container \"7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875\": container with ID starting with 7c2dd8fd154844d7d7c75439429bfb2d9d8488c2234061a63464f095c6011875 not found: ID does not exist" Jul 06 19:22:41.456717 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.456679 2571 scope.go:117] "RemoveContainer" containerID="e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae" Jul 06 19:22:41.456898 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.456882 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae"} err="failed to get container status \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": rpc error: code = NotFound desc = could not find container \"e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae\": container with ID starting with e208bb25788115100efa0f83b2503fdd03bbb31f8e2c84ccf97338978b65b1ae not found: ID does not exist" Jul 06 19:22:41.456940 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.456898 2571 scope.go:117] "RemoveContainer" containerID="ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6" Jul 06 19:22:41.457160 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.457128 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6"} err="failed to get container status \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": rpc error: code = NotFound desc = could not find container \"ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6\": container with ID starting with ad0ead5fa4baff74d345a3ad5d15e713a198ef8b5bc36eb3294813716f6e73c6 not found: ID does not exist" Jul 06 19:22:41.457160 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.457158 2571 scope.go:117] "RemoveContainer" containerID="abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3" Jul 06 19:22:41.457399 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.457376 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3"} err="failed to get container status \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": rpc error: code = NotFound desc = could not find container \"abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3\": container with ID starting with abbc928e40d5ae62133f1c0736c9112f8b7150c806727f963c5c8ad59cfdebe3 not found: ID does not exist" Jul 06 19:22:41.457444 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.457403 2571 scope.go:117] "RemoveContainer" containerID="2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7" Jul 06 19:22:41.457599 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.457582 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7"} err="failed to get container status \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": rpc error: code = NotFound desc = could not find container \"2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7\": container with ID starting with 2436e96ce5bae901dcc27ca1c2fb2dfa3b85d18109e8ed80048dd4cc9c938bb7 not found: ID does not exist" Jul 06 19:22:41.457649 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.457600 2571 scope.go:117] "RemoveContainer" containerID="9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061" Jul 06 19:22:41.457868 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.457850 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061"} err="failed to get container status \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": rpc error: code = NotFound desc = could not find container \"9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061\": container with ID starting with 9d54ca6b287eace0a35281deaa0fb8c85bacab474ff86e39137e161def0ab061 not found: ID does not exist" Jul 06 19:22:41.457939 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.457870 2571 scope.go:117] "RemoveContainer" containerID="fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e" Jul 06 19:22:41.458099 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.458082 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e"} err="failed to get container status \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": rpc error: code = NotFound desc = could not find container \"fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e\": container with ID starting with fb5c114c9d2ce68c2811ace716fe5548cbd474b2682c7060e160f8cd8c65373e not found: ID does not exist" Jul 06 19:22:41.461933 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.461914 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" path="/var/lib/kubelet/pods/a8c1f96f-e470-4e61-a477-263bd51a7840/volumes" Jul 06 19:22:41.462412 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462399 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:22:41.462679 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462669 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="thanos-sidecar" Jul 06 19:22:41.462713 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462682 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="thanos-sidecar" Jul 06 19:22:41.462713 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462694 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="init-config-reloader" Jul 06 19:22:41.462713 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462699 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="init-config-reloader" Jul 06 19:22:41.462713 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462707 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy-thanos" Jul 06 19:22:41.462713 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462712 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy-thanos" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462721 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy-web" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462726 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy-web" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462735 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="prometheus" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462739 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="prometheus" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462750 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="config-reloader" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462756 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="config-reloader" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462761 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462766 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462809 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="prometheus" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462818 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462824 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy-thanos" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462829 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="thanos-sidecar" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462835 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="config-reloader" Jul 06 19:22:41.462848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.462841 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c1f96f-e470-4e61-a477-263bd51a7840" containerName="kube-rbac-proxy-web" Jul 06 19:22:41.466910 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.466891 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.472812 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.469820 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 19:22:41.472812 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.470531 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 19:22:41.473163 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.473133 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qbnwq\"" Jul 06 19:22:41.473163 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.473137 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 19:22:41.473450 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.473432 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 19:22:41.473527 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.473486 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 19:22:41.475812 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.474207 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 19:22:41.475812 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.474392 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 19:22:41.475812 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.474588 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3u80smmpugct5\"" Jul 06 19:22:41.475812 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.475128 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 19:22:41.475812 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.475607 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 19:22:41.476138 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.475837 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 19:22:41.477000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.476977 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 19:22:41.477090 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.477078 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 19:22:41.478192 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.478172 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 19:22:41.480019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.480003 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 19:22:41.488603 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.488574 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:22:41.590811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590731 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.590811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590771 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.590811 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590835 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590863 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f310f78-62c3-47fa-81a1-bfb782e93d1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590883 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590909 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f310f78-62c3-47fa-81a1-bfb782e93d1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590957 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.590986 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591016 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591240 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591061 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591240 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591104 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-config\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591240 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591133 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591240 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591150 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591240 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591170 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591240 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591216 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591460 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591250 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591460 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591300 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.591460 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.591324 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nks\" (UniqueName: \"kubernetes.io/projected/8f310f78-62c3-47fa-81a1-bfb782e93d1c-kube-api-access-l5nks\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.692781 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.692748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.692781 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.692791 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.692813 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.692836 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.692863 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f310f78-62c3-47fa-81a1-bfb782e93d1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.692940 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.692986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.693027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f310f78-62c3-47fa-81a1-bfb782e93d1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.693058 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693517 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.693091 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693517 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.693142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693517 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.693171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693833 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.693597 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.693975 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.693956 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.694183 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.694156 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.696376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696236 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.696376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696341 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.696547 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696388 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-config\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.696547 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696430 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.696653 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696594 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f310f78-62c3-47fa-81a1-bfb782e93d1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697118 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697118 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696818 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697118 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696852 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697118 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696892 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697118 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696929 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697118 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696958 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nks\" (UniqueName: \"kubernetes.io/projected/8f310f78-62c3-47fa-81a1-bfb782e93d1c-kube-api-access-l5nks\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697118 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.696964 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f310f78-62c3-47fa-81a1-bfb782e93d1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697538 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.697236 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697598 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.697550 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.697873 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.697716 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.698185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.698126 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.698503 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.698479 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.698952 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.698927 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.699527 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.699502 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f310f78-62c3-47fa-81a1-bfb782e93d1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.699616 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.699550 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.700363 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.700341 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.700630 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.700607 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.700728 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.700665 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.701502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.701480 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f310f78-62c3-47fa-81a1-bfb782e93d1c-config\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.709609 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.709584 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nks\" (UniqueName: \"kubernetes.io/projected/8f310f78-62c3-47fa-81a1-bfb782e93d1c-kube-api-access-l5nks\") pod \"prometheus-k8s-0\" (UID: \"8f310f78-62c3-47fa-81a1-bfb782e93d1c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.783345 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.783307 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:41.932716 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:41.932682 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:22:41.935494 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:22:41.935469 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f310f78_62c3_47fa_81a1_bfb782e93d1c.slice/crio-441f9c5076bf1db257d42edea5e396c98cde3c3730e7fb6fa82edf020f36ff78 WatchSource:0}: Error finding container 441f9c5076bf1db257d42edea5e396c98cde3c3730e7fb6fa82edf020f36ff78: Status 404 returned error can't find the container with id 441f9c5076bf1db257d42edea5e396c98cde3c3730e7fb6fa82edf020f36ff78 Jul 06 19:22:42.402557 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:42.402519 2571 generic.go:356] "Generic (PLEG): container finished" podID="8f310f78-62c3-47fa-81a1-bfb782e93d1c" containerID="81e375851f50999317d195ec27c5d7cab65c70103162c3f5e3d96e291590e58e" exitCode=0 Jul 06 19:22:42.403020 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:42.402601 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f310f78-62c3-47fa-81a1-bfb782e93d1c","Type":"ContainerDied","Data":"81e375851f50999317d195ec27c5d7cab65c70103162c3f5e3d96e291590e58e"} Jul 06 19:22:42.403020 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:42.402668 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f310f78-62c3-47fa-81a1-bfb782e93d1c","Type":"ContainerStarted","Data":"441f9c5076bf1db257d42edea5e396c98cde3c3730e7fb6fa82edf020f36ff78"} Jul 06 19:22:43.409797 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.409761 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" event={"ID":"088fd4a9-6953-4809-bc48-d98f20202fe8","Type":"ContainerStarted","Data":"45d34bb79c7f0b048de0238a86513641757cfc74252107de25040e3b9dfa10e7"} Jul 06 19:22:43.410240 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.409876 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" event={"ID":"088fd4a9-6953-4809-bc48-d98f20202fe8","Type":"ContainerStarted","Data":"23c8e27d8e8a4386bb43e4380d29c7625fc17920756801708deba9e96dfa28fa"} Jul 06 19:22:43.410240 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.409913 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" event={"ID":"088fd4a9-6953-4809-bc48-d98f20202fe8","Type":"ContainerStarted","Data":"5e920ffdcafb807b625f3bd5277ba322766dda6f216e0c79dbd49a82cb440fbc"} Jul 06 19:22:43.412800 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.412782 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f310f78-62c3-47fa-81a1-bfb782e93d1c","Type":"ContainerStarted","Data":"78c688cb6271ce38b1ddedeff1f5cea6e345a7b2174d32579f08b56da93fc2cb"} Jul 06 19:22:43.412855 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.412838 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f310f78-62c3-47fa-81a1-bfb782e93d1c","Type":"ContainerStarted","Data":"926dbd8fb5f0dc642d7835fee30e2a388752eec9f9be883134ccdb398f49e27e"} Jul 06 19:22:43.412894 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.412866 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f310f78-62c3-47fa-81a1-bfb782e93d1c","Type":"ContainerStarted","Data":"0274a4085acc73a5256ae7ebd23d123ea61c3633c1ca72daca0a02abc8d501d2"} Jul 06 19:22:43.412931 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.412890 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f310f78-62c3-47fa-81a1-bfb782e93d1c","Type":"ContainerStarted","Data":"22b7f73a72e09c560ad4b2b078392ecb23f6692ecf651ce37b38a058f7dc70cf"} Jul 06 19:22:43.412931 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.412917 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f310f78-62c3-47fa-81a1-bfb782e93d1c","Type":"ContainerStarted","Data":"28e08c4ee8af36139d43df8fdb5698283b3c3cbba3e67dfb5625b9ed07911063"} Jul 06 19:22:43.412996 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.412941 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f310f78-62c3-47fa-81a1-bfb782e93d1c","Type":"ContainerStarted","Data":"64c584d5876f87b0585b22f02d00c57106df14ec3b5fd84dbb8032941889bb88"} Jul 06 19:22:43.433204 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.433155 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-64bd95bbdb-gs77h" podStartSLOduration=1.076417048 podStartE2EDuration="3.433139379s" totalImagesPullingTime="2.356722331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:43.432050561 +0000 UTC m=+150.492397488" watchObservedRunningTime="2026-07-06 19:22:43.433139379 +0000 UTC m=+150.493486306" Jul 06 19:22:43.461162 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:43.460854 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.437606063 podStartE2EDuration="2.460835316s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.229253ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:43.45943138 +0000 UTC m=+150.519778306" watchObservedRunningTime="2026-07-06 19:22:43.460835316 +0000 UTC m=+150.521182243" Jul 06 19:22:44.241549 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.241502 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-788fdd98c9-pgxps"] Jul 06 19:22:44.246069 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.246043 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.256932 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.256907 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788fdd98c9-pgxps"] Jul 06 19:22:44.428567 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.428534 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-trusted-ca-bundle\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.428567 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.428574 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-serving-cert\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.429008 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.428607 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-config\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.429008 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.428628 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-oauth-config\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.429008 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.428647 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-oauth-serving-cert\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.429008 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.428670 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjp9x\" (UniqueName: \"kubernetes.io/projected/30b68aab-a3bb-4689-87ce-f3409ff61eb2-kube-api-access-mjp9x\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.429008 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.428695 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-service-ca\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.529547 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.529508 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-serving-cert\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.529738 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.529610 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-config\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.529800 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.529751 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-oauth-config\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.529971 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.529946 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-oauth-serving-cert\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.530204 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.530182 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjp9x\" (UniqueName: \"kubernetes.io/projected/30b68aab-a3bb-4689-87ce-f3409ff61eb2-kube-api-access-mjp9x\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.530457 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.530409 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-config\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.532496 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.530720 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-service-ca\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.532496 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.530829 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-oauth-serving-cert\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.532496 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.531828 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-trusted-ca-bundle\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.532496 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.531991 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-service-ca\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.533043 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.532994 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-trusted-ca-bundle\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.533864 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.533483 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-serving-cert\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.534530 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.534509 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-oauth-config\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.539470 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.539428 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjp9x\" (UniqueName: \"kubernetes.io/projected/30b68aab-a3bb-4689-87ce-f3409ff61eb2-kube-api-access-mjp9x\") pod \"console-788fdd98c9-pgxps\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.556586 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.556551 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:44.692637 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:44.692586 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788fdd98c9-pgxps"] Jul 06 19:22:44.694849 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:22:44.694824 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b68aab_a3bb_4689_87ce_f3409ff61eb2.slice/crio-44c2dccd297d2a9b6974700e3a8ae3cf32cabc25f2b143a4e443a37874848761 WatchSource:0}: Error finding container 44c2dccd297d2a9b6974700e3a8ae3cf32cabc25f2b143a4e443a37874848761: Status 404 returned error can't find the container with id 44c2dccd297d2a9b6974700e3a8ae3cf32cabc25f2b143a4e443a37874848761 Jul 06 19:22:45.424416 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:45.424378 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788fdd98c9-pgxps" event={"ID":"30b68aab-a3bb-4689-87ce-f3409ff61eb2","Type":"ContainerStarted","Data":"1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de"} Jul 06 19:22:45.424610 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:45.424425 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788fdd98c9-pgxps" event={"ID":"30b68aab-a3bb-4689-87ce-f3409ff61eb2","Type":"ContainerStarted","Data":"44c2dccd297d2a9b6974700e3a8ae3cf32cabc25f2b143a4e443a37874848761"} Jul 06 19:22:45.442755 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:45.442704 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-788fdd98c9-pgxps" podStartSLOduration=1.442687225 podStartE2EDuration="1.442687225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:45.441988254 +0000 UTC m=+152.502335182" watchObservedRunningTime="2026-07-06 19:22:45.442687225 +0000 UTC m=+152.503034203" Jul 06 19:22:46.784525 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:46.784477 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:22:54.557354 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:54.557307 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:54.557830 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:54.557381 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:54.562149 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:54.562128 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:55.470068 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:55.470037 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:22:55.571823 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:22:55.571790 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd6c87777-zp7wc"] Jul 06 19:23:20.591918 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.591854 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5cd6c87777-zp7wc" podUID="c32d39d7-ceae-487d-816a-65f8d80d2478" containerName="console" containerID="cri-o://f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913" gracePeriod=15 Jul 06 19:23:20.856464 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.856432 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:23:20.948100 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948065 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-trusted-ca-bundle\") pod \"c32d39d7-ceae-487d-816a-65f8d80d2478\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " Jul 06 19:23:20.948100 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948106 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-oauth-config\") pod \"c32d39d7-ceae-487d-816a-65f8d80d2478\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " Jul 06 19:23:20.948368 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948162 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-serving-cert\") pod \"c32d39d7-ceae-487d-816a-65f8d80d2478\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " Jul 06 19:23:20.948368 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948197 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxzkj\" (UniqueName: \"kubernetes.io/projected/c32d39d7-ceae-487d-816a-65f8d80d2478-kube-api-access-rxzkj\") pod \"c32d39d7-ceae-487d-816a-65f8d80d2478\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " Jul 06 19:23:20.948453 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948358 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-console-config\") pod \"c32d39d7-ceae-487d-816a-65f8d80d2478\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " Jul 06 19:23:20.948453 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948407 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-service-ca\") pod \"c32d39d7-ceae-487d-816a-65f8d80d2478\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " Jul 06 19:23:20.948453 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948435 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-oauth-serving-cert\") pod \"c32d39d7-ceae-487d-816a-65f8d80d2478\" (UID: \"c32d39d7-ceae-487d-816a-65f8d80d2478\") " Jul 06 19:23:20.948627 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948496 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c32d39d7-ceae-487d-816a-65f8d80d2478" (UID: "c32d39d7-ceae-487d-816a-65f8d80d2478"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:23:20.948716 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948693 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-console-config" (OuterVolumeSpecName: "console-config") pod "c32d39d7-ceae-487d-816a-65f8d80d2478" (UID: "c32d39d7-ceae-487d-816a-65f8d80d2478"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:23:20.948870 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948845 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-service-ca" (OuterVolumeSpecName: "service-ca") pod "c32d39d7-ceae-487d-816a-65f8d80d2478" (UID: "c32d39d7-ceae-487d-816a-65f8d80d2478"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:23:20.948954 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948875 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-console-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:23:20.948954 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948891 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c32d39d7-ceae-487d-816a-65f8d80d2478" (UID: "c32d39d7-ceae-487d-816a-65f8d80d2478"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:23:20.948954 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.948897 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-trusted-ca-bundle\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:23:20.950607 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.950581 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c32d39d7-ceae-487d-816a-65f8d80d2478" (UID: "c32d39d7-ceae-487d-816a-65f8d80d2478"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:23:20.951055 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.951032 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c32d39d7-ceae-487d-816a-65f8d80d2478" (UID: "c32d39d7-ceae-487d-816a-65f8d80d2478"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:23:20.951104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:20.951068 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32d39d7-ceae-487d-816a-65f8d80d2478-kube-api-access-rxzkj" (OuterVolumeSpecName: "kube-api-access-rxzkj") pod "c32d39d7-ceae-487d-816a-65f8d80d2478" (UID: "c32d39d7-ceae-487d-816a-65f8d80d2478"). InnerVolumeSpecName "kube-api-access-rxzkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:23:21.050110 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.050065 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-oauth-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:23:21.050110 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.050106 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c32d39d7-ceae-487d-816a-65f8d80d2478-console-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:23:21.050110 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.050119 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzkj\" (UniqueName: \"kubernetes.io/projected/c32d39d7-ceae-487d-816a-65f8d80d2478-kube-api-access-rxzkj\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:23:21.050356 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.050132 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-service-ca\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:23:21.050356 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.050146 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c32d39d7-ceae-487d-816a-65f8d80d2478-oauth-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:23:21.579385 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.579349 2571 generic.go:356] "Generic (PLEG): container finished" podID="c32d39d7-ceae-487d-816a-65f8d80d2478" containerID="f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913" exitCode=0 Jul 06 19:23:21.579580 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.579401 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6c87777-zp7wc" Jul 06 19:23:21.579580 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.579433 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6c87777-zp7wc" event={"ID":"c32d39d7-ceae-487d-816a-65f8d80d2478","Type":"ContainerDied","Data":"f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913"} Jul 06 19:23:21.579580 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.579489 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6c87777-zp7wc" event={"ID":"c32d39d7-ceae-487d-816a-65f8d80d2478","Type":"ContainerDied","Data":"3ad05cf5235f77d4416eecd070cec71013a4409bf186c0f272c60da986a0f728"} Jul 06 19:23:21.579580 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.579514 2571 scope.go:117] "RemoveContainer" containerID="f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913" Jul 06 19:23:21.587720 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.587704 2571 scope.go:117] "RemoveContainer" containerID="f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913" Jul 06 19:23:21.587975 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:23:21.587958 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913\": container with ID starting with f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913 not found: ID does not exist" containerID="f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913" Jul 06 19:23:21.588038 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.587985 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913"} err="failed to get container status \"f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913\": rpc error: code = NotFound desc = could not find container \"f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913\": container with ID starting with f65d0f3e3d6c623bb26d19f11e5949ffbf246e614db945e259ed8099e85fd913 not found: ID does not exist" Jul 06 19:23:21.595123 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.595095 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd6c87777-zp7wc"] Jul 06 19:23:21.598616 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:21.598597 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cd6c87777-zp7wc"] Jul 06 19:23:23.460399 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:23.460368 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c32d39d7-ceae-487d-816a-65f8d80d2478" path="/var/lib/kubelet/pods/c32d39d7-ceae-487d-816a-65f8d80d2478/volumes" Jul 06 19:23:41.784029 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:41.783985 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:23:41.799879 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:41.799854 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:23:42.682245 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:42.682220 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:23:51.270572 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.270539 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cb7859556-pbzxg"] Jul 06 19:23:51.270944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.270853 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32d39d7-ceae-487d-816a-65f8d80d2478" containerName="console" Jul 06 19:23:51.270944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.270864 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32d39d7-ceae-487d-816a-65f8d80d2478" containerName="console" Jul 06 19:23:51.270944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.270924 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32d39d7-ceae-487d-816a-65f8d80d2478" containerName="console" Jul 06 19:23:51.275237 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.275220 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.291778 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.291756 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cb7859556-pbzxg"] Jul 06 19:23:51.310226 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.310206 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzqx\" (UniqueName: \"kubernetes.io/projected/01b72e62-d48f-4387-be9b-0e291c21b924-kube-api-access-8wzqx\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.310341 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.310254 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-oauth-config\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.310341 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.310300 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-serving-cert\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.310341 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.310333 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-trusted-ca-bundle\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.310459 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.310353 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-console-config\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.310459 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.310423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-service-ca\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.310459 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.310450 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-oauth-serving-cert\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.410801 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.410769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-oauth-config\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.410801 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.410803 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-serving-cert\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.411009 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.410830 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-trusted-ca-bundle\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.411009 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.410858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-console-config\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.411009 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.410902 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-service-ca\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.411009 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.410925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-oauth-serving-cert\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.411189 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.411057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzqx\" (UniqueName: \"kubernetes.io/projected/01b72e62-d48f-4387-be9b-0e291c21b924-kube-api-access-8wzqx\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.411665 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.411643 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-service-ca\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.411760 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.411703 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-console-config\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.411981 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.411752 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-oauth-serving-cert\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.412094 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.412066 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-trusted-ca-bundle\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.413419 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.413398 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-oauth-config\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.413567 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.413551 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-serving-cert\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.419808 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.419788 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzqx\" (UniqueName: \"kubernetes.io/projected/01b72e62-d48f-4387-be9b-0e291c21b924-kube-api-access-8wzqx\") pod \"console-6cb7859556-pbzxg\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.584384 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.584300 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:23:51.710536 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:51.710511 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cb7859556-pbzxg"] Jul 06 19:23:51.712980 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:23:51.712950 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b72e62_d48f_4387_be9b_0e291c21b924.slice/crio-87984b01146975e79bfbe48dd5ddbc4f0acfc00c108977531c4685c021c944e7 WatchSource:0}: Error finding container 87984b01146975e79bfbe48dd5ddbc4f0acfc00c108977531c4685c021c944e7: Status 404 returned error can't find the container with id 87984b01146975e79bfbe48dd5ddbc4f0acfc00c108977531c4685c021c944e7 Jul 06 19:23:52.704566 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:52.704531 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb7859556-pbzxg" event={"ID":"01b72e62-d48f-4387-be9b-0e291c21b924","Type":"ContainerStarted","Data":"ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda"} Jul 06 19:23:52.704952 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:52.704574 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb7859556-pbzxg" event={"ID":"01b72e62-d48f-4387-be9b-0e291c21b924","Type":"ContainerStarted","Data":"87984b01146975e79bfbe48dd5ddbc4f0acfc00c108977531c4685c021c944e7"} Jul 06 19:23:52.724063 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:23:52.724012 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cb7859556-pbzxg" podStartSLOduration=1.7239958469999999 podStartE2EDuration="1.723995847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:52.722806631 +0000 UTC m=+219.783153557" watchObservedRunningTime="2026-07-06 19:23:52.723995847 +0000 UTC m=+219.784342773" Jul 06 19:24:01.585214 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:01.585175 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:24:01.585620 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:01.585246 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:24:01.589904 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:01.589881 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:24:01.745802 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:01.745779 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:24:01.792365 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:01.792332 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-788fdd98c9-pgxps"] Jul 06 19:24:26.813674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:26.813619 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-788fdd98c9-pgxps" podUID="30b68aab-a3bb-4689-87ce-f3409ff61eb2" containerName="console" containerID="cri-o://1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de" gracePeriod=15 Jul 06 19:24:27.053231 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.053209 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:24:27.220486 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.220395 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjp9x\" (UniqueName: \"kubernetes.io/projected/30b68aab-a3bb-4689-87ce-f3409ff61eb2-kube-api-access-mjp9x\") pod \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " Jul 06 19:24:27.220486 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.220439 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-oauth-serving-cert\") pod \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " Jul 06 19:24:27.220486 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.220465 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-serving-cert\") pod \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " Jul 06 19:24:27.220747 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.220504 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-oauth-config\") pod \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " Jul 06 19:24:27.220747 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.220540 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-config\") pod \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " Jul 06 19:24:27.220747 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.220618 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-service-ca\") pod \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " Jul 06 19:24:27.220747 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.220659 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-trusted-ca-bundle\") pod \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\" (UID: \"30b68aab-a3bb-4689-87ce-f3409ff61eb2\") " Jul 06 19:24:27.220999 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.220969 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "30b68aab-a3bb-4689-87ce-f3409ff61eb2" (UID: "30b68aab-a3bb-4689-87ce-f3409ff61eb2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:27.221090 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.221067 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-service-ca" (OuterVolumeSpecName: "service-ca") pod "30b68aab-a3bb-4689-87ce-f3409ff61eb2" (UID: "30b68aab-a3bb-4689-87ce-f3409ff61eb2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:27.221149 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.221077 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-config" (OuterVolumeSpecName: "console-config") pod "30b68aab-a3bb-4689-87ce-f3409ff61eb2" (UID: "30b68aab-a3bb-4689-87ce-f3409ff61eb2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:27.221234 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.221215 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "30b68aab-a3bb-4689-87ce-f3409ff61eb2" (UID: "30b68aab-a3bb-4689-87ce-f3409ff61eb2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:27.222914 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.222896 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "30b68aab-a3bb-4689-87ce-f3409ff61eb2" (UID: "30b68aab-a3bb-4689-87ce-f3409ff61eb2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:24:27.223201 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.223183 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b68aab-a3bb-4689-87ce-f3409ff61eb2-kube-api-access-mjp9x" (OuterVolumeSpecName: "kube-api-access-mjp9x") pod "30b68aab-a3bb-4689-87ce-f3409ff61eb2" (UID: "30b68aab-a3bb-4689-87ce-f3409ff61eb2"). InnerVolumeSpecName "kube-api-access-mjp9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:27.223201 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.223189 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "30b68aab-a3bb-4689-87ce-f3409ff61eb2" (UID: "30b68aab-a3bb-4689-87ce-f3409ff61eb2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:24:27.321870 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.321844 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-service-ca\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:24:27.321870 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.321866 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-trusted-ca-bundle\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:24:27.322018 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.321876 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjp9x\" (UniqueName: \"kubernetes.io/projected/30b68aab-a3bb-4689-87ce-f3409ff61eb2-kube-api-access-mjp9x\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:24:27.322018 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.321885 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-oauth-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:24:27.322018 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.321894 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:24:27.322018 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.321902 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-oauth-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:24:27.322018 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.321910 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30b68aab-a3bb-4689-87ce-f3409ff61eb2-console-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:24:27.847933 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.847896 2571 generic.go:356] "Generic (PLEG): container finished" podID="30b68aab-a3bb-4689-87ce-f3409ff61eb2" containerID="1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de" exitCode=0 Jul 06 19:24:27.848372 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.847944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788fdd98c9-pgxps" event={"ID":"30b68aab-a3bb-4689-87ce-f3409ff61eb2","Type":"ContainerDied","Data":"1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de"} Jul 06 19:24:27.848372 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.847949 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788fdd98c9-pgxps" Jul 06 19:24:27.848372 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.848019 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788fdd98c9-pgxps" event={"ID":"30b68aab-a3bb-4689-87ce-f3409ff61eb2","Type":"ContainerDied","Data":"44c2dccd297d2a9b6974700e3a8ae3cf32cabc25f2b143a4e443a37874848761"} Jul 06 19:24:27.848372 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.848044 2571 scope.go:117] "RemoveContainer" containerID="1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de" Jul 06 19:24:27.856094 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.856078 2571 scope.go:117] "RemoveContainer" containerID="1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de" Jul 06 19:24:27.856422 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:24:27.856403 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de\": container with ID starting with 1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de not found: ID does not exist" containerID="1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de" Jul 06 19:24:27.856488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.856432 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de"} err="failed to get container status \"1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de\": rpc error: code = NotFound desc = could not find container \"1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de\": container with ID starting with 1a65c8852d2591a07f1359d5d8a2e2e141a028aa13cdbabb8164859c689be6de not found: ID does not exist" Jul 06 19:24:27.866188 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.866164 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-788fdd98c9-pgxps"] Jul 06 19:24:27.870734 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:27.870713 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-788fdd98c9-pgxps"] Jul 06 19:24:29.460573 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:24:29.460543 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30b68aab-a3bb-4689-87ce-f3409ff61eb2" path="/var/lib/kubelet/pods/30b68aab-a3bb-4689-87ce-f3409ff61eb2/volumes" Jul 06 19:25:04.564005 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.563972 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-slml9"] Jul 06 19:25:04.564486 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.564356 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b68aab-a3bb-4689-87ce-f3409ff61eb2" containerName="console" Jul 06 19:25:04.564486 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.564367 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b68aab-a3bb-4689-87ce-f3409ff61eb2" containerName="console" Jul 06 19:25:04.564486 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.564420 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b68aab-a3bb-4689-87ce-f3409ff61eb2" containerName="console" Jul 06 19:25:04.567214 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.567199 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" Jul 06 19:25:04.570332 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.570310 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:25:04.571260 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.571241 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k7mp6\"" Jul 06 19:25:04.571361 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.571261 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:25:04.587540 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.587519 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-slml9"] Jul 06 19:25:04.618778 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.618751 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsz5b\" (UniqueName: \"kubernetes.io/projected/29e72f68-55ce-4fdc-9924-3054bc789664-kube-api-access-dsz5b\") pod \"cert-manager-cainjector-559cdd8c87-slml9\" (UID: \"29e72f68-55ce-4fdc-9924-3054bc789664\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" Jul 06 19:25:04.618881 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.618853 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29e72f68-55ce-4fdc-9924-3054bc789664-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-slml9\" (UID: \"29e72f68-55ce-4fdc-9924-3054bc789664\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" Jul 06 19:25:04.719591 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.719570 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29e72f68-55ce-4fdc-9924-3054bc789664-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-slml9\" (UID: \"29e72f68-55ce-4fdc-9924-3054bc789664\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" Jul 06 19:25:04.719686 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.719607 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsz5b\" (UniqueName: \"kubernetes.io/projected/29e72f68-55ce-4fdc-9924-3054bc789664-kube-api-access-dsz5b\") pod \"cert-manager-cainjector-559cdd8c87-slml9\" (UID: \"29e72f68-55ce-4fdc-9924-3054bc789664\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" Jul 06 19:25:04.727599 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.727570 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsz5b\" (UniqueName: \"kubernetes.io/projected/29e72f68-55ce-4fdc-9924-3054bc789664-kube-api-access-dsz5b\") pod \"cert-manager-cainjector-559cdd8c87-slml9\" (UID: \"29e72f68-55ce-4fdc-9924-3054bc789664\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" Jul 06 19:25:04.727599 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.727582 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29e72f68-55ce-4fdc-9924-3054bc789664-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-slml9\" (UID: \"29e72f68-55ce-4fdc-9924-3054bc789664\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" Jul 06 19:25:04.887870 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:04.887788 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" Jul 06 19:25:05.017307 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:05.017264 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-slml9"] Jul 06 19:25:05.019695 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:25:05.019652 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29e72f68_55ce_4fdc_9924_3054bc789664.slice/crio-01d7dd5eb3c3e49668b5472d28531e2511787eaf4f2f33ed7eb88d23311e0dc9 WatchSource:0}: Error finding container 01d7dd5eb3c3e49668b5472d28531e2511787eaf4f2f33ed7eb88d23311e0dc9: Status 404 returned error can't find the container with id 01d7dd5eb3c3e49668b5472d28531e2511787eaf4f2f33ed7eb88d23311e0dc9 Jul 06 19:25:06.003405 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:06.003354 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" event={"ID":"29e72f68-55ce-4fdc-9924-3054bc789664","Type":"ContainerStarted","Data":"01d7dd5eb3c3e49668b5472d28531e2511787eaf4f2f33ed7eb88d23311e0dc9"} Jul 06 19:25:09.019297 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:09.019248 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" event={"ID":"29e72f68-55ce-4fdc-9924-3054bc789664","Type":"ContainerStarted","Data":"3cfff64b923514b8fbe825f80074d381b40583b3da84ecdc1f88841b4eed2c36"} Jul 06 19:25:09.039640 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:09.039584 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-slml9" podStartSLOduration=1.6921294709999999 podStartE2EDuration="5.039568787s" totalImagesPullingTime="3.347439316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:09.037333517 +0000 UTC m=+296.097680443" watchObservedRunningTime="2026-07-06 19:25:09.039568787 +0000 UTC m=+296.099915712" Jul 06 19:25:13.095117 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.095081 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz"] Jul 06 19:25:13.098637 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.098616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" Jul 06 19:25:13.104254 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.104234 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-b456l\"" Jul 06 19:25:13.105254 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.105221 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:25:13.105408 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.105384 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:25:13.108701 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.108682 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz"] Jul 06 19:25:13.193751 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.193718 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84bed43d-3933-42f6-b76d-7803a642380e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bp2lz\" (UID: \"84bed43d-3933-42f6-b76d-7803a642380e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" Jul 06 19:25:13.193917 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.193757 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5qf\" (UniqueName: \"kubernetes.io/projected/84bed43d-3933-42f6-b76d-7803a642380e-kube-api-access-qb5qf\") pod \"openshift-lws-operator-fd8ccff4c-bp2lz\" (UID: \"84bed43d-3933-42f6-b76d-7803a642380e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" Jul 06 19:25:13.294213 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.294183 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5qf\" (UniqueName: \"kubernetes.io/projected/84bed43d-3933-42f6-b76d-7803a642380e-kube-api-access-qb5qf\") pod \"openshift-lws-operator-fd8ccff4c-bp2lz\" (UID: \"84bed43d-3933-42f6-b76d-7803a642380e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" Jul 06 19:25:13.294560 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.294540 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84bed43d-3933-42f6-b76d-7803a642380e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bp2lz\" (UID: \"84bed43d-3933-42f6-b76d-7803a642380e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" Jul 06 19:25:13.294928 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.294901 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84bed43d-3933-42f6-b76d-7803a642380e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bp2lz\" (UID: \"84bed43d-3933-42f6-b76d-7803a642380e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" Jul 06 19:25:13.307604 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.307586 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:25:13.310289 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.310254 2571 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:25:13.318203 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.318187 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:25:13.328674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.328651 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5qf\" (UniqueName: \"kubernetes.io/projected/84bed43d-3933-42f6-b76d-7803a642380e-kube-api-access-qb5qf\") pod \"openshift-lws-operator-fd8ccff4c-bp2lz\" (UID: \"84bed43d-3933-42f6-b76d-7803a642380e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" Jul 06 19:25:13.410432 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.410374 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-b456l\"" Jul 06 19:25:13.418207 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.418186 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" Jul 06 19:25:13.555592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:13.555570 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz"] Jul 06 19:25:13.557849 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:25:13.557824 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84bed43d_3933_42f6_b76d_7803a642380e.slice/crio-f2b76df77742bf9f37ce1cd46bd5836873e5d3717ccd891aaaeb12b4d36967a9 WatchSource:0}: Error finding container f2b76df77742bf9f37ce1cd46bd5836873e5d3717ccd891aaaeb12b4d36967a9: Status 404 returned error can't find the container with id f2b76df77742bf9f37ce1cd46bd5836873e5d3717ccd891aaaeb12b4d36967a9 Jul 06 19:25:14.040144 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:14.040112 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" event={"ID":"84bed43d-3933-42f6-b76d-7803a642380e","Type":"ContainerStarted","Data":"f2b76df77742bf9f37ce1cd46bd5836873e5d3717ccd891aaaeb12b4d36967a9"} Jul 06 19:25:17.057232 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:17.057201 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" event={"ID":"84bed43d-3933-42f6-b76d-7803a642380e","Type":"ContainerStarted","Data":"a3ce26e9313020db01794fe02e383e8bfb9d04dea0140ee3cf31c0c42f015d51"} Jul 06 19:25:17.077898 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:17.077855 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bp2lz" podStartSLOduration=1.644425837 podStartE2EDuration="4.077838565s" totalImagesPullingTime="2.433412728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:17.076760644 +0000 UTC m=+304.137107572" watchObservedRunningTime="2026-07-06 19:25:17.077838565 +0000 UTC m=+304.138185491" Jul 06 19:25:33.059356 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.059326 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm"] Jul 06 19:25:33.065674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.065657 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.069853 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.069832 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tfnth\"" Jul 06 19:25:33.069966 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.069876 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:25:33.070397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.070369 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:25:33.070545 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.070530 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:25:33.070629 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.070535 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:25:33.095641 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.095617 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm"] Jul 06 19:25:33.156719 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.156691 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdf137ce-5324-4116-906d-a8ee2b92c24b-tmp\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.156867 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.156722 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvmq\" (UniqueName: \"kubernetes.io/projected/cdf137ce-5324-4116-906d-a8ee2b92c24b-kube-api-access-sqvmq\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.156867 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.156769 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cdf137ce-5324-4116-906d-a8ee2b92c24b-manifests\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.156867 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.156795 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdf137ce-5324-4116-906d-a8ee2b92c24b-webhook-cert\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.156867 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.156812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cdf137ce-5324-4116-906d-a8ee2b92c24b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.257519 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.257492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdf137ce-5324-4116-906d-a8ee2b92c24b-tmp\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.257519 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.257519 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvmq\" (UniqueName: \"kubernetes.io/projected/cdf137ce-5324-4116-906d-a8ee2b92c24b-kube-api-access-sqvmq\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.257724 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.257560 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cdf137ce-5324-4116-906d-a8ee2b92c24b-manifests\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.257724 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.257590 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdf137ce-5324-4116-906d-a8ee2b92c24b-webhook-cert\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.257724 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.257687 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cdf137ce-5324-4116-906d-a8ee2b92c24b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.257984 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.257967 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cdf137ce-5324-4116-906d-a8ee2b92c24b-manifests\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.259855 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.259827 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdf137ce-5324-4116-906d-a8ee2b92c24b-tmp\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.260132 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.260113 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cdf137ce-5324-4116-906d-a8ee2b92c24b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.260201 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.260158 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdf137ce-5324-4116-906d-a8ee2b92c24b-webhook-cert\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.273933 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.273910 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvmq\" (UniqueName: \"kubernetes.io/projected/cdf137ce-5324-4116-906d-a8ee2b92c24b-kube-api-access-sqvmq\") pod \"opendatahub-operator-controller-manager-69fd78d8d5-7bxvm\" (UID: \"cdf137ce-5324-4116-906d-a8ee2b92c24b\") " pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.376641 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.376573 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:33.528635 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.528607 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm"] Jul 06 19:25:33.530684 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:25:33.530654 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf137ce_5324_4116_906d_a8ee2b92c24b.slice/crio-17032189259e326a39ce34cf6267e5a40822050beee26c925cfd5b4b714c8626 WatchSource:0}: Error finding container 17032189259e326a39ce34cf6267e5a40822050beee26c925cfd5b4b714c8626: Status 404 returned error can't find the container with id 17032189259e326a39ce34cf6267e5a40822050beee26c925cfd5b4b714c8626 Jul 06 19:25:33.532406 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:33.532391 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:25:34.129185 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:34.129150 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" event={"ID":"cdf137ce-5324-4116-906d-a8ee2b92c24b","Type":"ContainerStarted","Data":"17032189259e326a39ce34cf6267e5a40822050beee26c925cfd5b4b714c8626"} Jul 06 19:25:38.150945 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:38.150906 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" event={"ID":"cdf137ce-5324-4116-906d-a8ee2b92c24b","Type":"ContainerStarted","Data":"f697f9c3dabc080297359b0c46af14913d83549ec0c3332fd701faae72e2bbe8"} Jul 06 19:25:39.156776 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:39.156742 2571 generic.go:356] "Generic (PLEG): container finished" podID="cdf137ce-5324-4116-906d-a8ee2b92c24b" containerID="f697f9c3dabc080297359b0c46af14913d83549ec0c3332fd701faae72e2bbe8" exitCode=0 Jul 06 19:25:39.157315 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:39.156821 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" event={"ID":"cdf137ce-5324-4116-906d-a8ee2b92c24b","Type":"ContainerDied","Data":"f697f9c3dabc080297359b0c46af14913d83549ec0c3332fd701faae72e2bbe8"} Jul 06 19:25:40.163242 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:40.163208 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" event={"ID":"cdf137ce-5324-4116-906d-a8ee2b92c24b","Type":"ContainerStarted","Data":"1559ef2d9f971b174beeee57c7083b06cbb6ad63386d2952fb8503a6195fc515"} Jul 06 19:25:40.163669 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:40.163310 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:25:40.194667 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:40.194620 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" podStartSLOduration=1.836609862 podStartE2EDuration="7.194604156s" totalImagesPullingTime="4.295999156s" totalInitContainerRuntime="1.061995138s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:40.193506286 +0000 UTC m=+327.253853223" watchObservedRunningTime="2026-07-06 19:25:40.194604156 +0000 UTC m=+327.254951081" Jul 06 19:25:51.175251 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:25:51.175218 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69fd78d8d5-7bxvm" Jul 06 19:26:03.923977 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:03.923897 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq"] Jul 06 19:26:03.927558 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:03.927523 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:03.933416 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:03.933392 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:26:03.933558 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:03.933392 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-n7cxb\"" Jul 06 19:26:03.933775 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:03.933752 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:26:03.934285 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:03.934245 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:26:03.948873 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:03.948850 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq"] Jul 06 19:26:04.032056 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.032013 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-metrics-cert\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.032213 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.032072 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkgc9\" (UniqueName: \"kubernetes.io/projected/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-kube-api-access-kkgc9\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.032213 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.032158 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-manager-config\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.032342 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.032210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-cert\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.133667 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.133626 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-manager-config\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.133864 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.133695 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-cert\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.133864 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.133769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-metrics-cert\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.133864 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.133815 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkgc9\" (UniqueName: \"kubernetes.io/projected/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-kube-api-access-kkgc9\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.134621 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.134571 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-manager-config\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.136458 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.136429 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-cert\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.136458 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.136441 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-metrics-cert\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.150663 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.150616 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkgc9\" (UniqueName: \"kubernetes.io/projected/1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa-kube-api-access-kkgc9\") pod \"lws-controller-manager-7d885dd8f9-xg5sq\" (UID: \"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa\") " pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.239166 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.239094 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:04.407840 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:04.407815 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq"] Jul 06 19:26:04.409421 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:26:04.409397 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9f93d7_7753_451d_b29b_7d3ebcf1b9fa.slice/crio-692368f53eba77067b734198cd9e75a4ee139822911cf9be454318816f24f27a WatchSource:0}: Error finding container 692368f53eba77067b734198cd9e75a4ee139822911cf9be454318816f24f27a: Status 404 returned error can't find the container with id 692368f53eba77067b734198cd9e75a4ee139822911cf9be454318816f24f27a Jul 06 19:26:05.284377 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:05.284335 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" event={"ID":"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa","Type":"ContainerStarted","Data":"692368f53eba77067b734198cd9e75a4ee139822911cf9be454318816f24f27a"} Jul 06 19:26:06.442789 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.442753 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2xttg"] Jul 06 19:26:06.446653 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.446634 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:06.448969 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.448939 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll"] Jul 06 19:26:06.449152 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.449133 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 19:26:06.449208 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.449155 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-nc5ts\"" Jul 06 19:26:06.452683 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.452665 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:06.454987 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.454949 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 19:26:06.455181 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.455153 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-b4ffs\"" Jul 06 19:26:06.461485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.461464 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2xttg"] Jul 06 19:26:06.467249 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.467227 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll"] Jul 06 19:26:06.555612 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.555574 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4694615b-ae46-44cd-8a55-2f1b46d595ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-2xttg\" (UID: \"4694615b-ae46-44cd-8a55-2f1b46d595ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:06.555786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.555655 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2ac511-e78f-4dfc-adb9-205defb7f9f5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6ll\" (UID: \"7d2ac511-e78f-4dfc-adb9-205defb7f9f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:06.555786 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.555685 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfps9\" (UniqueName: \"kubernetes.io/projected/4694615b-ae46-44cd-8a55-2f1b46d595ee-kube-api-access-pfps9\") pod \"kserve-controller-manager-d6f4b65b8-2xttg\" (UID: \"4694615b-ae46-44cd-8a55-2f1b46d595ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:06.555877 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.555784 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnmw2\" (UniqueName: \"kubernetes.io/projected/7d2ac511-e78f-4dfc-adb9-205defb7f9f5-kube-api-access-hnmw2\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6ll\" (UID: \"7d2ac511-e78f-4dfc-adb9-205defb7f9f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:06.656713 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.656678 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnmw2\" (UniqueName: \"kubernetes.io/projected/7d2ac511-e78f-4dfc-adb9-205defb7f9f5-kube-api-access-hnmw2\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6ll\" (UID: \"7d2ac511-e78f-4dfc-adb9-205defb7f9f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:06.656877 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.656728 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4694615b-ae46-44cd-8a55-2f1b46d595ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-2xttg\" (UID: \"4694615b-ae46-44cd-8a55-2f1b46d595ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:06.656877 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.656763 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2ac511-e78f-4dfc-adb9-205defb7f9f5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6ll\" (UID: \"7d2ac511-e78f-4dfc-adb9-205defb7f9f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:06.656877 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.656781 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfps9\" (UniqueName: \"kubernetes.io/projected/4694615b-ae46-44cd-8a55-2f1b46d595ee-kube-api-access-pfps9\") pod \"kserve-controller-manager-d6f4b65b8-2xttg\" (UID: \"4694615b-ae46-44cd-8a55-2f1b46d595ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:06.657054 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:26:06.656915 2571 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 19:26:06.657054 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:26:06.657026 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d2ac511-e78f-4dfc-adb9-205defb7f9f5-cert podName:7d2ac511-e78f-4dfc-adb9-205defb7f9f5 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:07.157002023 +0000 UTC m=+354.217348942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7d2ac511-e78f-4dfc-adb9-205defb7f9f5-cert") pod "llmisvc-controller-manager-777c6bd7d5-bq6ll" (UID: "7d2ac511-e78f-4dfc-adb9-205defb7f9f5") : secret "llmisvc-webhook-server-cert" not found Jul 06 19:26:06.659354 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.659330 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4694615b-ae46-44cd-8a55-2f1b46d595ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-2xttg\" (UID: \"4694615b-ae46-44cd-8a55-2f1b46d595ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:06.669626 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.669605 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfps9\" (UniqueName: \"kubernetes.io/projected/4694615b-ae46-44cd-8a55-2f1b46d595ee-kube-api-access-pfps9\") pod \"kserve-controller-manager-d6f4b65b8-2xttg\" (UID: \"4694615b-ae46-44cd-8a55-2f1b46d595ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:06.684744 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.684720 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnmw2\" (UniqueName: \"kubernetes.io/projected/7d2ac511-e78f-4dfc-adb9-205defb7f9f5-kube-api-access-hnmw2\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6ll\" (UID: \"7d2ac511-e78f-4dfc-adb9-205defb7f9f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:06.758359 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.758264 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:06.903491 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:06.903447 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2xttg"] Jul 06 19:26:06.905981 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:26:06.905951 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4694615b_ae46_44cd_8a55_2f1b46d595ee.slice/crio-56f6a3774faba012ee62dc37701490e32ecedb3da48c47bd5dac0983b58f0822 WatchSource:0}: Error finding container 56f6a3774faba012ee62dc37701490e32ecedb3da48c47bd5dac0983b58f0822: Status 404 returned error can't find the container with id 56f6a3774faba012ee62dc37701490e32ecedb3da48c47bd5dac0983b58f0822 Jul 06 19:26:07.168286 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:07.168244 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2ac511-e78f-4dfc-adb9-205defb7f9f5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6ll\" (UID: \"7d2ac511-e78f-4dfc-adb9-205defb7f9f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:07.170950 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:07.170919 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2ac511-e78f-4dfc-adb9-205defb7f9f5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6ll\" (UID: \"7d2ac511-e78f-4dfc-adb9-205defb7f9f5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:07.295022 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:07.294984 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" event={"ID":"4694615b-ae46-44cd-8a55-2f1b46d595ee","Type":"ContainerStarted","Data":"56f6a3774faba012ee62dc37701490e32ecedb3da48c47bd5dac0983b58f0822"} Jul 06 19:26:07.366242 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:07.366211 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:07.532429 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:07.532397 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll"] Jul 06 19:26:07.884450 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:26:07.884420 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7d2ac511_e78f_4dfc_adb9_205defb7f9f5.slice/crio-b168de5e377e94581c7b31edfdbbfde663affa1eb25ba12ab2029d2a706f3f6a WatchSource:0}: Error finding container b168de5e377e94581c7b31edfdbbfde663affa1eb25ba12ab2029d2a706f3f6a: Status 404 returned error can't find the container with id b168de5e377e94581c7b31edfdbbfde663affa1eb25ba12ab2029d2a706f3f6a Jul 06 19:26:08.300583 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:08.300548 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" event={"ID":"7d2ac511-e78f-4dfc-adb9-205defb7f9f5","Type":"ContainerStarted","Data":"b168de5e377e94581c7b31edfdbbfde663affa1eb25ba12ab2029d2a706f3f6a"} Jul 06 19:26:08.301925 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:08.301905 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" event={"ID":"1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa","Type":"ContainerStarted","Data":"d080a265ca72428cc4212398303593a4c87407807fbaa36d764937d51d73de33"} Jul 06 19:26:08.302045 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:08.302026 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:08.344291 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:08.344235 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" podStartSLOduration=1.807189714 podStartE2EDuration="5.344219478s" totalImagesPullingTime="3.537029764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:08.341542737 +0000 UTC m=+355.401889665" watchObservedRunningTime="2026-07-06 19:26:08.344219478 +0000 UTC m=+355.404566404" Jul 06 19:26:10.315753 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:10.315720 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" event={"ID":"4694615b-ae46-44cd-8a55-2f1b46d595ee","Type":"ContainerStarted","Data":"926a256831ebd9c31eb283f39f0e3bf62ce01dd64105c4a4eb5687258a858df5"} Jul 06 19:26:10.316224 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:10.315779 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:10.337256 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:10.337185 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" podStartSLOduration=1.299001804 podStartE2EDuration="4.337169233s" totalImagesPullingTime="3.038167429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:10.335468941 +0000 UTC m=+357.395815867" watchObservedRunningTime="2026-07-06 19:26:10.337169233 +0000 UTC m=+357.397516159" Jul 06 19:26:12.327203 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:12.327164 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" event={"ID":"7d2ac511-e78f-4dfc-adb9-205defb7f9f5","Type":"ContainerStarted","Data":"f435f0ae8bfd0c994d9fb66a5b5b18c910ed6884ceb94f616535cd5bdc20dba7"} Jul 06 19:26:12.327602 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:12.327256 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:12.346971 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:12.346922 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" podStartSLOduration=2.738499219 podStartE2EDuration="6.346907759s" totalImagesPullingTime="3.60840854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:12.344323294 +0000 UTC m=+359.404670220" watchObservedRunningTime="2026-07-06 19:26:12.346907759 +0000 UTC m=+359.407254720" Jul 06 19:26:19.313024 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:19.312992 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d885dd8f9-xg5sq" Jul 06 19:26:41.326790 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:41.326752 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xttg" Jul 06 19:26:43.334484 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:43.334452 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6ll" Jul 06 19:26:54.506303 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.506251 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzq6d"] Jul 06 19:26:54.511167 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.511147 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jzq6d" Jul 06 19:26:54.514640 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.514619 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:26:54.514909 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.514887 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:26:54.515037 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.514962 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-x2vn8\"" Jul 06 19:26:54.537926 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.537900 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzq6d"] Jul 06 19:26:54.690168 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.690136 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fm6d\" (UniqueName: \"kubernetes.io/projected/41162583-218e-4cea-9fbb-911ee27fe11d-kube-api-access-2fm6d\") pod \"kuadrant-operator-catalog-jzq6d\" (UID: \"41162583-218e-4cea-9fbb-911ee27fe11d\") " pod="kuadrant-system/kuadrant-operator-catalog-jzq6d" Jul 06 19:26:54.790616 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.790589 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fm6d\" (UniqueName: \"kubernetes.io/projected/41162583-218e-4cea-9fbb-911ee27fe11d-kube-api-access-2fm6d\") pod \"kuadrant-operator-catalog-jzq6d\" (UID: \"41162583-218e-4cea-9fbb-911ee27fe11d\") " pod="kuadrant-system/kuadrant-operator-catalog-jzq6d" Jul 06 19:26:54.791051 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.791025 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzq6d"] Jul 06 19:26:54.791337 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:26:54.791316 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2fm6d], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-catalog-jzq6d" podUID="41162583-218e-4cea-9fbb-911ee27fe11d" Jul 06 19:26:54.799304 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:54.799264 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fm6d\" (UniqueName: \"kubernetes.io/projected/41162583-218e-4cea-9fbb-911ee27fe11d-kube-api-access-2fm6d\") pod \"kuadrant-operator-catalog-jzq6d\" (UID: \"41162583-218e-4cea-9fbb-911ee27fe11d\") " pod="kuadrant-system/kuadrant-operator-catalog-jzq6d" Jul 06 19:26:55.008589 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.008557 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cxrmx"] Jul 06 19:26:55.011023 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.011008 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" Jul 06 19:26:55.022309 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.022264 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cxrmx"] Jul 06 19:26:55.193987 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.193906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glfng\" (UniqueName: \"kubernetes.io/projected/9947ecb8-5a9d-401a-80be-8cc3fb3e7e73-kube-api-access-glfng\") pod \"kuadrant-operator-catalog-cxrmx\" (UID: \"9947ecb8-5a9d-401a-80be-8cc3fb3e7e73\") " pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" Jul 06 19:26:55.295001 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.294971 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glfng\" (UniqueName: \"kubernetes.io/projected/9947ecb8-5a9d-401a-80be-8cc3fb3e7e73-kube-api-access-glfng\") pod \"kuadrant-operator-catalog-cxrmx\" (UID: \"9947ecb8-5a9d-401a-80be-8cc3fb3e7e73\") " pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" Jul 06 19:26:55.303970 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.303946 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glfng\" (UniqueName: \"kubernetes.io/projected/9947ecb8-5a9d-401a-80be-8cc3fb3e7e73-kube-api-access-glfng\") pod \"kuadrant-operator-catalog-cxrmx\" (UID: \"9947ecb8-5a9d-401a-80be-8cc3fb3e7e73\") " pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" Jul 06 19:26:55.319978 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.319953 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" Jul 06 19:26:55.471418 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.471362 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cxrmx"] Jul 06 19:26:55.474080 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:26:55.474053 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9947ecb8_5a9d_401a_80be_8cc3fb3e7e73.slice/crio-259a1e6a040290b8e231a8951e23d40267fac8edfc44de93f8b2c0e2e69412f6 WatchSource:0}: Error finding container 259a1e6a040290b8e231a8951e23d40267fac8edfc44de93f8b2c0e2e69412f6: Status 404 returned error can't find the container with id 259a1e6a040290b8e231a8951e23d40267fac8edfc44de93f8b2c0e2e69412f6 Jul 06 19:26:55.522159 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.522129 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" event={"ID":"9947ecb8-5a9d-401a-80be-8cc3fb3e7e73","Type":"ContainerStarted","Data":"259a1e6a040290b8e231a8951e23d40267fac8edfc44de93f8b2c0e2e69412f6"} Jul 06 19:26:55.522519 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.522174 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jzq6d" Jul 06 19:26:55.528801 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.528782 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jzq6d" Jul 06 19:26:55.699216 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.699189 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fm6d\" (UniqueName: \"kubernetes.io/projected/41162583-218e-4cea-9fbb-911ee27fe11d-kube-api-access-2fm6d\") pod \"41162583-218e-4cea-9fbb-911ee27fe11d\" (UID: \"41162583-218e-4cea-9fbb-911ee27fe11d\") " Jul 06 19:26:55.701370 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.701348 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41162583-218e-4cea-9fbb-911ee27fe11d-kube-api-access-2fm6d" (OuterVolumeSpecName: "kube-api-access-2fm6d") pod "41162583-218e-4cea-9fbb-911ee27fe11d" (UID: "41162583-218e-4cea-9fbb-911ee27fe11d"). InnerVolumeSpecName "kube-api-access-2fm6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:26:55.800218 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:55.800183 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fm6d\" (UniqueName: \"kubernetes.io/projected/41162583-218e-4cea-9fbb-911ee27fe11d-kube-api-access-2fm6d\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:26:56.526228 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:56.526195 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jzq6d" Jul 06 19:26:56.586525 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:56.586473 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzq6d"] Jul 06 19:26:56.589019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:56.588996 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzq6d"] Jul 06 19:26:57.461298 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:57.461237 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41162583-218e-4cea-9fbb-911ee27fe11d" path="/var/lib/kubelet/pods/41162583-218e-4cea-9fbb-911ee27fe11d/volumes" Jul 06 19:26:58.536223 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:58.536191 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" event={"ID":"9947ecb8-5a9d-401a-80be-8cc3fb3e7e73","Type":"ContainerStarted","Data":"bc1b7519c68dfde4e1b54e1b523b3dc54fdade9f3f0df7b8ea6c235d49ad991b"} Jul 06 19:26:58.561968 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:26:58.556644 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" podStartSLOduration=2.476625077 podStartE2EDuration="4.556625251s" totalImagesPullingTime="2.080000174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:58.554907072 +0000 UTC m=+405.615254012" watchObservedRunningTime="2026-07-06 19:26:58.556625251 +0000 UTC m=+405.616972178" Jul 06 19:27:05.320709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:05.320674 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" Jul 06 19:27:05.321133 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:05.320734 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" Jul 06 19:27:05.343352 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:05.343329 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" Jul 06 19:27:05.589594 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:05.589515 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-cxrmx" Jul 06 19:27:21.068930 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.068893 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm"] Jul 06 19:27:21.076628 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.076612 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:21.083087 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.082702 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pctjq\"" Jul 06 19:27:21.103186 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.103161 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm"] Jul 06 19:27:21.219282 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.219235 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2vb\" (UniqueName: \"kubernetes.io/projected/83d3ed59-27bb-46f5-8eff-eaf5c196e7c3-kube-api-access-sl2vb\") pod \"limitador-operator-controller-manager-859d7b78bd-j4bcm\" (UID: \"83d3ed59-27bb-46f5-8eff-eaf5c196e7c3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:21.320747 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.320671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2vb\" (UniqueName: \"kubernetes.io/projected/83d3ed59-27bb-46f5-8eff-eaf5c196e7c3-kube-api-access-sl2vb\") pod \"limitador-operator-controller-manager-859d7b78bd-j4bcm\" (UID: \"83d3ed59-27bb-46f5-8eff-eaf5c196e7c3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:21.349387 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.349356 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2vb\" (UniqueName: \"kubernetes.io/projected/83d3ed59-27bb-46f5-8eff-eaf5c196e7c3-kube-api-access-sl2vb\") pod \"limitador-operator-controller-manager-859d7b78bd-j4bcm\" (UID: \"83d3ed59-27bb-46f5-8eff-eaf5c196e7c3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:21.385932 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.385909 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:21.556265 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.556236 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm"] Jul 06 19:27:21.557201 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:27:21.557176 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d3ed59_27bb_46f5_8eff_eaf5c196e7c3.slice/crio-9da22f5dbdcde160c04ff872a9407136afe8fec7cbad260958407c284765ab41 WatchSource:0}: Error finding container 9da22f5dbdcde160c04ff872a9407136afe8fec7cbad260958407c284765ab41: Status 404 returned error can't find the container with id 9da22f5dbdcde160c04ff872a9407136afe8fec7cbad260958407c284765ab41 Jul 06 19:27:21.637164 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:21.637090 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" event={"ID":"83d3ed59-27bb-46f5-8eff-eaf5c196e7c3","Type":"ContainerStarted","Data":"9da22f5dbdcde160c04ff872a9407136afe8fec7cbad260958407c284765ab41"} Jul 06 19:27:23.648251 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:23.648164 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" event={"ID":"83d3ed59-27bb-46f5-8eff-eaf5c196e7c3","Type":"ContainerStarted","Data":"890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b"} Jul 06 19:27:23.648251 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:23.648238 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:23.676336 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:23.676288 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" podStartSLOduration=0.896918431 podStartE2EDuration="2.676254233s" totalImagesPullingTime="1.779335802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:23.675679179 +0000 UTC m=+430.736026103" watchObservedRunningTime="2026-07-06 19:27:23.676254233 +0000 UTC m=+430.736601159" Jul 06 19:27:31.244341 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:31.244301 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-959dj"] Jul 06 19:27:31.248665 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:31.248647 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" Jul 06 19:27:31.252520 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:31.252497 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zjgpq\"" Jul 06 19:27:31.350206 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:31.350168 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-959dj"] Jul 06 19:27:31.412488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:31.412453 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74bfq\" (UniqueName: \"kubernetes.io/projected/c02f3250-b02f-4b77-aa10-2d9d80a1298c-kube-api-access-74bfq\") pod \"authorino-operator-75cc8d5cb7-959dj\" (UID: \"c02f3250-b02f-4b77-aa10-2d9d80a1298c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" Jul 06 19:27:31.513452 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:31.513368 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74bfq\" (UniqueName: \"kubernetes.io/projected/c02f3250-b02f-4b77-aa10-2d9d80a1298c-kube-api-access-74bfq\") pod \"authorino-operator-75cc8d5cb7-959dj\" (UID: \"c02f3250-b02f-4b77-aa10-2d9d80a1298c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" Jul 06 19:27:31.532713 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:31.532684 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74bfq\" (UniqueName: \"kubernetes.io/projected/c02f3250-b02f-4b77-aa10-2d9d80a1298c-kube-api-access-74bfq\") pod \"authorino-operator-75cc8d5cb7-959dj\" (UID: \"c02f3250-b02f-4b77-aa10-2d9d80a1298c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" Jul 06 19:27:31.559473 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:31.559450 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" Jul 06 19:27:31.727424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:31.727313 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-959dj"] Jul 06 19:27:31.729730 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:27:31.729704 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02f3250_b02f_4b77_aa10_2d9d80a1298c.slice/crio-c13cd8fe6af4c4a31e5c6cbc172f3854747e1198fc64a0d6a740d2cc5bc350f3 WatchSource:0}: Error finding container c13cd8fe6af4c4a31e5c6cbc172f3854747e1198fc64a0d6a740d2cc5bc350f3: Status 404 returned error can't find the container with id c13cd8fe6af4c4a31e5c6cbc172f3854747e1198fc64a0d6a740d2cc5bc350f3 Jul 06 19:27:32.692763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:32.692724 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" event={"ID":"c02f3250-b02f-4b77-aa10-2d9d80a1298c","Type":"ContainerStarted","Data":"c13cd8fe6af4c4a31e5c6cbc172f3854747e1198fc64a0d6a740d2cc5bc350f3"} Jul 06 19:27:34.655415 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:34.655385 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:34.704800 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:34.704768 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" event={"ID":"c02f3250-b02f-4b77-aa10-2d9d80a1298c","Type":"ContainerStarted","Data":"f62cece7ea728d290aea3c56e0b3776f2be65bf8f831ee1825c517d39d695086"} Jul 06 19:27:34.704989 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:34.704868 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" Jul 06 19:27:34.737130 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:34.737071 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" podStartSLOduration=1.595517598 podStartE2EDuration="3.737025207s" totalImagesPullingTime="2.141507609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:34.734377547 +0000 UTC m=+441.794724472" watchObservedRunningTime="2026-07-06 19:27:34.737025207 +0000 UTC m=+441.797372134" Jul 06 19:27:45.712208 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:45.712181 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-959dj" Jul 06 19:27:49.994435 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:49.994401 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm"] Jul 06 19:27:49.994866 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:49.994651 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" containerName="manager" containerID="cri-o://890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b" gracePeriod=2 Jul 06 19:27:50.011056 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.011032 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm"] Jul 06 19:27:50.052880 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.052848 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv"] Jul 06 19:27:50.053325 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.053307 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" containerName="manager" Jul 06 19:27:50.053325 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.053325 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" containerName="manager" Jul 06 19:27:50.053497 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.053434 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" containerName="manager" Jul 06 19:27:50.056737 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.056716 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" Jul 06 19:27:50.059513 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:27:50.059474 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-j4bcm\" is forbidden: User \"system:node:ip-10-0-131-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-211.ec2.internal' and this object" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:50.062202 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:27:50.062178 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ggtmv\" is forbidden: User \"system:node:ip-10-0-131-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-211.ec2.internal' and this object" podUID="7d267ed6-3497-411c-a842-2f42cc120f8d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" Jul 06 19:27:50.080675 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.080652 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv"] Jul 06 19:27:50.175050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.175015 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhn6s\" (UniqueName: \"kubernetes.io/projected/7d267ed6-3497-411c-a842-2f42cc120f8d-kube-api-access-vhn6s\") pod \"limitador-operator-controller-manager-859d7b78bd-ggtmv\" (UID: \"7d267ed6-3497-411c-a842-2f42cc120f8d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" Jul 06 19:27:50.244182 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.244162 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:50.246446 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:27:50.246394 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-j4bcm\" is forbidden: User \"system:node:ip-10-0-131-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-211.ec2.internal' and this object" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:50.275873 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.275850 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl2vb\" (UniqueName: \"kubernetes.io/projected/83d3ed59-27bb-46f5-8eff-eaf5c196e7c3-kube-api-access-sl2vb\") pod \"83d3ed59-27bb-46f5-8eff-eaf5c196e7c3\" (UID: \"83d3ed59-27bb-46f5-8eff-eaf5c196e7c3\") " Jul 06 19:27:50.276090 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.276074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhn6s\" (UniqueName: \"kubernetes.io/projected/7d267ed6-3497-411c-a842-2f42cc120f8d-kube-api-access-vhn6s\") pod \"limitador-operator-controller-manager-859d7b78bd-ggtmv\" (UID: \"7d267ed6-3497-411c-a842-2f42cc120f8d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" Jul 06 19:27:50.278099 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.278075 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d3ed59-27bb-46f5-8eff-eaf5c196e7c3-kube-api-access-sl2vb" (OuterVolumeSpecName: "kube-api-access-sl2vb") pod "83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" (UID: "83d3ed59-27bb-46f5-8eff-eaf5c196e7c3"). InnerVolumeSpecName "kube-api-access-sl2vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:27:50.300534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.300513 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhn6s\" (UniqueName: \"kubernetes.io/projected/7d267ed6-3497-411c-a842-2f42cc120f8d-kube-api-access-vhn6s\") pod \"limitador-operator-controller-manager-859d7b78bd-ggtmv\" (UID: \"7d267ed6-3497-411c-a842-2f42cc120f8d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" Jul 06 19:27:50.371080 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.371055 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" Jul 06 19:27:50.377058 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.377026 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl2vb\" (UniqueName: \"kubernetes.io/projected/83d3ed59-27bb-46f5-8eff-eaf5c196e7c3-kube-api-access-sl2vb\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:27:50.523510 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.523484 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv"] Jul 06 19:27:50.526286 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:27:50.526246 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d267ed6_3497_411c_a842_2f42cc120f8d.slice/crio-4c9903f4002d8b70247f25e97beaf6e434444cd39207017e216c1e37f3d736ed WatchSource:0}: Error finding container 4c9903f4002d8b70247f25e97beaf6e434444cd39207017e216c1e37f3d736ed: Status 404 returned error can't find the container with id 4c9903f4002d8b70247f25e97beaf6e434444cd39207017e216c1e37f3d736ed Jul 06 19:27:50.781025 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.780931 2571 generic.go:356] "Generic (PLEG): container finished" podID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" containerID="890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b" exitCode=0 Jul 06 19:27:50.781025 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.780983 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:50.781248 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.781058 2571 scope.go:117] "RemoveContainer" containerID="890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b" Jul 06 19:27:50.782760 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.782734 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" event={"ID":"7d267ed6-3497-411c-a842-2f42cc120f8d","Type":"ContainerStarted","Data":"9294554a8a359738d7310cb1b86b03b9695d5f69d90219db2e5beeb3de7fc93c"} Jul 06 19:27:50.782865 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.782782 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" event={"ID":"7d267ed6-3497-411c-a842-2f42cc120f8d","Type":"ContainerStarted","Data":"4c9903f4002d8b70247f25e97beaf6e434444cd39207017e216c1e37f3d736ed"} Jul 06 19:27:50.782865 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.782818 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" Jul 06 19:27:50.784736 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:27:50.784706 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-j4bcm\" is forbidden: User \"system:node:ip-10-0-131-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-211.ec2.internal' and this object" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:50.787732 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:27:50.787698 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-j4bcm\" is forbidden: User \"system:node:ip-10-0-131-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-211.ec2.internal' and this object" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:50.790478 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.790459 2571 scope.go:117] "RemoveContainer" containerID="890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b" Jul 06 19:27:50.790780 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:27:50.790762 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b\": container with ID starting with 890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b not found: ID does not exist" containerID="890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b" Jul 06 19:27:50.790844 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.790789 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b"} err="failed to get container status \"890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b\": rpc error: code = NotFound desc = could not find container \"890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b\": container with ID starting with 890efd98c0a5ad1ada983ef0e3bc5a5c68da9f85c417d44d9b3f1d916f5f318b not found: ID does not exist" Jul 06 19:27:50.823354 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:50.823310 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" podStartSLOduration=0.82329791 podStartE2EDuration="823.29791ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:50.821452093 +0000 UTC m=+457.881799018" watchObservedRunningTime="2026-07-06 19:27:50.82329791 +0000 UTC m=+457.883644829" Jul 06 19:27:50.824917 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:27:50.824894 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-j4bcm\" is forbidden: User \"system:node:ip-10-0-131-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-211.ec2.internal' and this object" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j4bcm" Jul 06 19:27:51.460759 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:51.460726 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d3ed59-27bb-46f5-8eff-eaf5c196e7c3" path="/var/lib/kubelet/pods/83d3ed59-27bb-46f5-8eff-eaf5c196e7c3/volumes" Jul 06 19:27:52.194916 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.194885 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c"] Jul 06 19:27:52.199883 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.199865 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:27:52.202400 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.202384 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p7bkg\"" Jul 06 19:27:52.228912 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.228887 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c"] Jul 06 19:27:52.293674 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.293640 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f6881-12cd-4ff4-849d-b2c2182d219d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-prm5c\" (UID: \"e58f6881-12cd-4ff4-849d-b2c2182d219d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:27:52.293839 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.293687 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2fn\" (UniqueName: \"kubernetes.io/projected/e58f6881-12cd-4ff4-849d-b2c2182d219d-kube-api-access-jg2fn\") pod \"kuadrant-operator-controller-manager-86878c769f-prm5c\" (UID: \"e58f6881-12cd-4ff4-849d-b2c2182d219d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:27:52.394234 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.394203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f6881-12cd-4ff4-849d-b2c2182d219d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-prm5c\" (UID: \"e58f6881-12cd-4ff4-849d-b2c2182d219d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:27:52.394424 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.394248 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2fn\" (UniqueName: \"kubernetes.io/projected/e58f6881-12cd-4ff4-849d-b2c2182d219d-kube-api-access-jg2fn\") pod \"kuadrant-operator-controller-manager-86878c769f-prm5c\" (UID: \"e58f6881-12cd-4ff4-849d-b2c2182d219d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:27:52.394602 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.394581 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f6881-12cd-4ff4-849d-b2c2182d219d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-prm5c\" (UID: \"e58f6881-12cd-4ff4-849d-b2c2182d219d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:27:52.419629 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.419606 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2fn\" (UniqueName: \"kubernetes.io/projected/e58f6881-12cd-4ff4-849d-b2c2182d219d-kube-api-access-jg2fn\") pod \"kuadrant-operator-controller-manager-86878c769f-prm5c\" (UID: \"e58f6881-12cd-4ff4-849d-b2c2182d219d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:27:52.510066 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.509980 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:27:52.650134 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.650111 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c"] Jul 06 19:27:52.651907 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:27:52.651876 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58f6881_12cd_4ff4_849d_b2c2182d219d.slice/crio-f0d045eca5b32520cadcc65fa3623192ea93418bf0a3705992ab5fe51a5bab85 WatchSource:0}: Error finding container f0d045eca5b32520cadcc65fa3623192ea93418bf0a3705992ab5fe51a5bab85: Status 404 returned error can't find the container with id f0d045eca5b32520cadcc65fa3623192ea93418bf0a3705992ab5fe51a5bab85 Jul 06 19:27:52.795306 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:52.795259 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" event={"ID":"e58f6881-12cd-4ff4-849d-b2c2182d219d","Type":"ContainerStarted","Data":"f0d045eca5b32520cadcc65fa3623192ea93418bf0a3705992ab5fe51a5bab85"} Jul 06 19:27:54.065789 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.065752 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d4b949455-2qh74"] Jul 06 19:27:54.069875 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.069850 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.088988 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.088963 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4b949455-2qh74"] Jul 06 19:27:54.109171 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.109138 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce416fef-b171-432c-8931-936823a8529b-console-oauth-config\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.109378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.109199 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-service-ca\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.109378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.109228 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-trusted-ca-bundle\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.109378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.109309 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-oauth-serving-cert\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.109378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.109371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-console-config\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.109686 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.109490 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhrm\" (UniqueName: \"kubernetes.io/projected/ce416fef-b171-432c-8931-936823a8529b-kube-api-access-llhrm\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.109686 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.109550 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce416fef-b171-432c-8931-936823a8529b-console-serving-cert\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.210915 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.210870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-console-config\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.211052 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.210969 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhrm\" (UniqueName: \"kubernetes.io/projected/ce416fef-b171-432c-8931-936823a8529b-kube-api-access-llhrm\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.211052 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.211030 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce416fef-b171-432c-8931-936823a8529b-console-serving-cert\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.211160 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.211056 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce416fef-b171-432c-8931-936823a8529b-console-oauth-config\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.211160 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.211077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-service-ca\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.211160 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.211098 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-trusted-ca-bundle\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.211160 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.211135 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-oauth-serving-cert\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.211985 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.211914 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-console-config\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.211985 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.211940 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-service-ca\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.211985 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.211940 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-oauth-serving-cert\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.212326 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.212248 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce416fef-b171-432c-8931-936823a8529b-trusted-ca-bundle\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.213847 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.213825 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce416fef-b171-432c-8931-936823a8529b-console-serving-cert\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.213944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.213921 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce416fef-b171-432c-8931-936823a8529b-console-oauth-config\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.219759 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.219736 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhrm\" (UniqueName: \"kubernetes.io/projected/ce416fef-b171-432c-8931-936823a8529b-kube-api-access-llhrm\") pod \"console-7d4b949455-2qh74\" (UID: \"ce416fef-b171-432c-8931-936823a8529b\") " pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.382909 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.382822 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:27:54.535884 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.535859 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4b949455-2qh74"] Jul 06 19:27:54.538808 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:27:54.538783 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce416fef_b171_432c_8931_936823a8529b.slice/crio-98317c688a10a4b0e17b71b8fe5d1b090abb3cb036e6a06d926c8dc445147f89 WatchSource:0}: Error finding container 98317c688a10a4b0e17b71b8fe5d1b090abb3cb036e6a06d926c8dc445147f89: Status 404 returned error can't find the container with id 98317c688a10a4b0e17b71b8fe5d1b090abb3cb036e6a06d926c8dc445147f89 Jul 06 19:27:54.808841 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.808803 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b949455-2qh74" event={"ID":"ce416fef-b171-432c-8931-936823a8529b","Type":"ContainerStarted","Data":"a77dae397fdf9ad6bcf915c50b45bc9a18a39d25f89c706df03d95c1a00f956a"} Jul 06 19:27:54.809028 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.808873 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b949455-2qh74" event={"ID":"ce416fef-b171-432c-8931-936823a8529b","Type":"ContainerStarted","Data":"98317c688a10a4b0e17b71b8fe5d1b090abb3cb036e6a06d926c8dc445147f89"} Jul 06 19:27:54.834960 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:54.834875 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d4b949455-2qh74" podStartSLOduration=0.834853417 podStartE2EDuration="834.853417ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:54.833905518 +0000 UTC m=+461.894252440" watchObservedRunningTime="2026-07-06 19:27:54.834853417 +0000 UTC m=+461.895200344" Jul 06 19:27:57.827416 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:57.827377 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" event={"ID":"e58f6881-12cd-4ff4-849d-b2c2182d219d","Type":"ContainerStarted","Data":"69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4"} Jul 06 19:27:57.827821 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:57.827471 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:27:57.858864 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:27:57.858816 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" podStartSLOduration=0.901274652 podStartE2EDuration="5.858802487s" totalImagesPullingTime="4.957527835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:57.853885069 +0000 UTC m=+464.914231996" watchObservedRunningTime="2026-07-06 19:27:57.858802487 +0000 UTC m=+464.919149412" Jul 06 19:28:01.791682 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:01.791654 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ggtmv" Jul 06 19:28:04.383087 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:04.383059 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:28:04.383528 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:04.383157 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:28:04.389041 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:04.388448 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:28:04.865002 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:04.864976 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d4b949455-2qh74" Jul 06 19:28:04.919470 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:04.919440 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cb7859556-pbzxg"] Jul 06 19:28:08.835541 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:08.835513 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:28:29.940835 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:29.940794 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6cb7859556-pbzxg" podUID="01b72e62-d48f-4387-be9b-0e291c21b924" containerName="console" containerID="cri-o://ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda" gracePeriod=15 Jul 06 19:28:30.203001 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.202934 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:28:30.241818 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.241790 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-serving-cert\") pod \"01b72e62-d48f-4387-be9b-0e291c21b924\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " Jul 06 19:28:30.241974 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.241827 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-console-config\") pod \"01b72e62-d48f-4387-be9b-0e291c21b924\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " Jul 06 19:28:30.241974 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.241850 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wzqx\" (UniqueName: \"kubernetes.io/projected/01b72e62-d48f-4387-be9b-0e291c21b924-kube-api-access-8wzqx\") pod \"01b72e62-d48f-4387-be9b-0e291c21b924\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " Jul 06 19:28:30.241974 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.241880 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-oauth-config\") pod \"01b72e62-d48f-4387-be9b-0e291c21b924\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " Jul 06 19:28:30.241974 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.241902 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-oauth-serving-cert\") pod \"01b72e62-d48f-4387-be9b-0e291c21b924\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " Jul 06 19:28:30.241974 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.241951 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-service-ca\") pod \"01b72e62-d48f-4387-be9b-0e291c21b924\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " Jul 06 19:28:30.242243 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.242224 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-console-config" (OuterVolumeSpecName: "console-config") pod "01b72e62-d48f-4387-be9b-0e291c21b924" (UID: "01b72e62-d48f-4387-be9b-0e291c21b924"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:28:30.242378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.242351 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-console-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:28:30.242572 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.242542 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01b72e62-d48f-4387-be9b-0e291c21b924" (UID: "01b72e62-d48f-4387-be9b-0e291c21b924"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:28:30.242648 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.242569 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-service-ca" (OuterVolumeSpecName: "service-ca") pod "01b72e62-d48f-4387-be9b-0e291c21b924" (UID: "01b72e62-d48f-4387-be9b-0e291c21b924"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:28:30.244257 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.244224 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01b72e62-d48f-4387-be9b-0e291c21b924" (UID: "01b72e62-d48f-4387-be9b-0e291c21b924"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:28:30.244667 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.244640 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01b72e62-d48f-4387-be9b-0e291c21b924" (UID: "01b72e62-d48f-4387-be9b-0e291c21b924"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:28:30.244667 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.244655 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b72e62-d48f-4387-be9b-0e291c21b924-kube-api-access-8wzqx" (OuterVolumeSpecName: "kube-api-access-8wzqx") pod "01b72e62-d48f-4387-be9b-0e291c21b924" (UID: "01b72e62-d48f-4387-be9b-0e291c21b924"). InnerVolumeSpecName "kube-api-access-8wzqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:28:30.343557 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.343514 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-trusted-ca-bundle\") pod \"01b72e62-d48f-4387-be9b-0e291c21b924\" (UID: \"01b72e62-d48f-4387-be9b-0e291c21b924\") " Jul 06 19:28:30.343720 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.343683 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-service-ca\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:28:30.343720 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.343694 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:28:30.343720 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.343704 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzqx\" (UniqueName: \"kubernetes.io/projected/01b72e62-d48f-4387-be9b-0e291c21b924-kube-api-access-8wzqx\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:28:30.343720 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.343712 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01b72e62-d48f-4387-be9b-0e291c21b924-console-oauth-config\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:28:30.343850 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.343722 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-oauth-serving-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:28:30.343887 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.343858 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "01b72e62-d48f-4387-be9b-0e291c21b924" (UID: "01b72e62-d48f-4387-be9b-0e291c21b924"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:28:30.445133 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.445102 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b72e62-d48f-4387-be9b-0e291c21b924-trusted-ca-bundle\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:28:30.989494 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.989458 2571 generic.go:356] "Generic (PLEG): container finished" podID="01b72e62-d48f-4387-be9b-0e291c21b924" containerID="ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda" exitCode=0 Jul 06 19:28:30.989878 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.989507 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb7859556-pbzxg" Jul 06 19:28:30.989878 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.989525 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb7859556-pbzxg" event={"ID":"01b72e62-d48f-4387-be9b-0e291c21b924","Type":"ContainerDied","Data":"ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda"} Jul 06 19:28:30.989878 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.989570 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb7859556-pbzxg" event={"ID":"01b72e62-d48f-4387-be9b-0e291c21b924","Type":"ContainerDied","Data":"87984b01146975e79bfbe48dd5ddbc4f0acfc00c108977531c4685c021c944e7"} Jul 06 19:28:30.989878 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.989597 2571 scope.go:117] "RemoveContainer" containerID="ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda" Jul 06 19:28:30.998681 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.998663 2571 scope.go:117] "RemoveContainer" containerID="ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda" Jul 06 19:28:30.998949 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:28:30.998930 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda\": container with ID starting with ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda not found: ID does not exist" containerID="ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda" Jul 06 19:28:30.998996 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:30.998958 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda"} err="failed to get container status \"ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda\": rpc error: code = NotFound desc = could not find container \"ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda\": container with ID starting with ebdbe7502c98f31b20b1f318778edf907a8d1a9680406f3f5ba1731286544dda not found: ID does not exist" Jul 06 19:28:31.012506 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:31.012471 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cb7859556-pbzxg"] Jul 06 19:28:31.021190 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:31.021167 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cb7859556-pbzxg"] Jul 06 19:28:31.460943 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:28:31.460905 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b72e62-d48f-4387-be9b-0e291c21b924" path="/var/lib/kubelet/pods/01b72e62-d48f-4387-be9b-0e291c21b924/volumes" Jul 06 19:29:52.383578 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:29:52.383545 2571 generic.go:356] "Generic (PLEG): container finished" podID="e58f6881-12cd-4ff4-849d-b2c2182d219d" containerID="69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4" exitCode=0 Jul 06 19:29:52.384125 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:29:52.383624 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" event={"ID":"e58f6881-12cd-4ff4-849d-b2c2182d219d","Type":"ContainerDied","Data":"69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4"} Jul 06 19:29:52.384125 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:29:52.384039 2571 scope.go:117] "RemoveContainer" containerID="69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4" Jul 06 19:29:52.510618 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:29:52.510587 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:29:53.390757 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:29:53.390722 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" event={"ID":"e58f6881-12cd-4ff4-849d-b2c2182d219d","Type":"ContainerStarted","Data":"ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7"} Jul 06 19:29:53.391152 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:29:53.390826 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:30:04.397697 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:04.397662 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:30:55.053956 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.053881 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls"] Jul 06 19:30:55.055471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.054349 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b72e62-d48f-4387-be9b-0e291c21b924" containerName="console" Jul 06 19:30:55.055471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.054365 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b72e62-d48f-4387-be9b-0e291c21b924" containerName="console" Jul 06 19:30:55.055471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.054444 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b72e62-d48f-4387-be9b-0e291c21b924" containerName="console" Jul 06 19:30:55.056350 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.056336 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.060156 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.060133 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-msnlq\"" Jul 06 19:30:55.060697 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.060678 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:30:55.060829 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.060721 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 19:30:55.061216 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.061200 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:30:55.083025 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.082993 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls"] Jul 06 19:30:55.184413 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.184376 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.184577 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.184433 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.184619 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.184550 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.184659 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.184631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.184694 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.184665 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.184738 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.184692 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv948\" (UniqueName: \"kubernetes.io/projected/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-kube-api-access-hv948\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.285653 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.285618 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.285653 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.285662 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.285946 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.285679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.285946 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.285696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv948\" (UniqueName: \"kubernetes.io/projected/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-kube-api-access-hv948\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.285946 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.285775 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.285946 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.285804 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.286337 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.286302 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.286337 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.286331 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.286476 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.286375 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.288137 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.288112 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.288375 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.288358 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.294182 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.294158 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv948\" (UniqueName: \"kubernetes.io/projected/4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe-kube-api-access-hv948\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6dkls\" (UID: \"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.368521 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.368435 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:30:55.508913 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.508888 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls"] Jul 06 19:30:55.514729 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:30:55.514695 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcf5aba_77ef_42a0_bfe4_0f9fbd759dfe.slice/crio-8cc0a73d25162a690e5fd08a0c23b57015f50c190b26a287cbc155b2f542299c WatchSource:0}: Error finding container 8cc0a73d25162a690e5fd08a0c23b57015f50c190b26a287cbc155b2f542299c: Status 404 returned error can't find the container with id 8cc0a73d25162a690e5fd08a0c23b57015f50c190b26a287cbc155b2f542299c Jul 06 19:30:55.516289 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.516255 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:30:55.697318 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:55.697213 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" event={"ID":"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe","Type":"ContainerStarted","Data":"8cc0a73d25162a690e5fd08a0c23b57015f50c190b26a287cbc155b2f542299c"} Jul 06 19:30:56.704815 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:56.704780 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" event={"ID":"4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe","Type":"ContainerStarted","Data":"35381f41d889514cd9e16af40661723dcfa4cb152ab7e331bb8849e268ded7d0"} Jul 06 19:30:56.725230 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:30:56.725171 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" podStartSLOduration=0.726862364 podStartE2EDuration="1.7251533s" totalImagesPullingTime="998.290936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:56.723234604 +0000 UTC m=+643.783581532" watchObservedRunningTime="2026-07-06 19:30:56.7251533 +0000 UTC m=+643.785500226" Jul 06 19:31:05.368697 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:05.368654 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:31:05.369147 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:05.368729 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:31:05.381990 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:05.381958 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:31:05.760804 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:05.760723 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6dkls" Jul 06 19:31:06.335555 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.335520 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc"] Jul 06 19:31:06.338425 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.338405 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.341560 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.341541 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 19:31:06.349289 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.349247 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc"] Jul 06 19:31:06.497131 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.497085 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.497131 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.497144 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwb5b\" (UniqueName: \"kubernetes.io/projected/13d46d60-dc29-41fc-9004-e931dccd63c3-kube-api-access-nwb5b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.497633 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.497167 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.497633 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.497242 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.497633 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.497311 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.497633 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.497326 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d46d60-dc29-41fc-9004-e931dccd63c3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.598634 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.598523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.598634 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.598601 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwb5b\" (UniqueName: \"kubernetes.io/projected/13d46d60-dc29-41fc-9004-e931dccd63c3-kube-api-access-nwb5b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.598898 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.598638 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.598898 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.598851 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.599016 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.598985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.599016 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.599009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d46d60-dc29-41fc-9004-e931dccd63c3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.599154 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.599128 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.599212 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.599161 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.599365 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.599347 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.601041 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.601018 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d46d60-dc29-41fc-9004-e931dccd63c3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.601629 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.601613 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d46d60-dc29-41fc-9004-e931dccd63c3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.607669 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.607648 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwb5b\" (UniqueName: \"kubernetes.io/projected/13d46d60-dc29-41fc-9004-e931dccd63c3-kube-api-access-nwb5b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc\" (UID: \"13d46d60-dc29-41fc-9004-e931dccd63c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.650131 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.650102 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:06.790199 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:06.790168 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc"] Jul 06 19:31:06.791759 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:31:06.791731 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d46d60_dc29_41fc_9004_e931dccd63c3.slice/crio-56afefd610c349544c32cf7f458bc7fe260606d2212fc3c3f37fbc3f41a2cd5f WatchSource:0}: Error finding container 56afefd610c349544c32cf7f458bc7fe260606d2212fc3c3f37fbc3f41a2cd5f: Status 404 returned error can't find the container with id 56afefd610c349544c32cf7f458bc7fe260606d2212fc3c3f37fbc3f41a2cd5f Jul 06 19:31:07.763149 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:07.763110 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" event={"ID":"13d46d60-dc29-41fc-9004-e931dccd63c3","Type":"ContainerStarted","Data":"5ec38f4f6e78d6ab126d21494c322b316cc28f97c3a940b64f5b09532d9ca63b"} Jul 06 19:31:07.763577 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:07.763179 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" event={"ID":"13d46d60-dc29-41fc-9004-e931dccd63c3","Type":"ContainerStarted","Data":"56afefd610c349544c32cf7f458bc7fe260606d2212fc3c3f37fbc3f41a2cd5f"} Jul 06 19:31:16.650921 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:16.650876 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:16.651464 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:16.650979 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:16.663451 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:16.663429 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:31:16.703325 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:16.703262 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" podStartSLOduration=10.541596611 podStartE2EDuration="10.703249367s" totalImagesPullingTime="161.652756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:31:07.797417217 +0000 UTC m=+654.857764144" watchObservedRunningTime="2026-07-06 19:31:16.703249367 +0000 UTC m=+663.763596292" Jul 06 19:31:16.823554 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:31:16.823527 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc" Jul 06 19:38:05.387190 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:05.387110 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c"] Jul 06 19:38:05.387775 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:05.387367 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" podUID="e58f6881-12cd-4ff4-849d-b2c2182d219d" containerName="manager" containerID="cri-o://ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7" gracePeriod=10 Jul 06 19:38:07.834630 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.834609 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:38:07.890169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.890139 2571 generic.go:356] "Generic (PLEG): container finished" podID="e58f6881-12cd-4ff4-849d-b2c2182d219d" containerID="ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7" exitCode=0 Jul 06 19:38:07.890376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.890180 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" event={"ID":"e58f6881-12cd-4ff4-849d-b2c2182d219d","Type":"ContainerDied","Data":"ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7"} Jul 06 19:38:07.890376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.890200 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" Jul 06 19:38:07.890376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.890243 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c" event={"ID":"e58f6881-12cd-4ff4-849d-b2c2182d219d","Type":"ContainerDied","Data":"f0d045eca5b32520cadcc65fa3623192ea93418bf0a3705992ab5fe51a5bab85"} Jul 06 19:38:07.890376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.890292 2571 scope.go:117] "RemoveContainer" containerID="ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7" Jul 06 19:38:07.899803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.899768 2571 scope.go:117] "RemoveContainer" containerID="69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4" Jul 06 19:38:07.905387 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.905366 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f6881-12cd-4ff4-849d-b2c2182d219d-extensions-socket-volume\") pod \"e58f6881-12cd-4ff4-849d-b2c2182d219d\" (UID: \"e58f6881-12cd-4ff4-849d-b2c2182d219d\") " Jul 06 19:38:07.905488 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.905476 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg2fn\" (UniqueName: \"kubernetes.io/projected/e58f6881-12cd-4ff4-849d-b2c2182d219d-kube-api-access-jg2fn\") pod \"e58f6881-12cd-4ff4-849d-b2c2182d219d\" (UID: \"e58f6881-12cd-4ff4-849d-b2c2182d219d\") " Jul 06 19:38:07.905707 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.905688 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58f6881-12cd-4ff4-849d-b2c2182d219d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e58f6881-12cd-4ff4-849d-b2c2182d219d" (UID: "e58f6881-12cd-4ff4-849d-b2c2182d219d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:38:07.907436 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.907412 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58f6881-12cd-4ff4-849d-b2c2182d219d-kube-api-access-jg2fn" (OuterVolumeSpecName: "kube-api-access-jg2fn") pod "e58f6881-12cd-4ff4-849d-b2c2182d219d" (UID: "e58f6881-12cd-4ff4-849d-b2c2182d219d"). InnerVolumeSpecName "kube-api-access-jg2fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:38:07.908252 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.908233 2571 scope.go:117] "RemoveContainer" containerID="ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7" Jul 06 19:38:07.908563 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:38:07.908544 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7\": container with ID starting with ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7 not found: ID does not exist" containerID="ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7" Jul 06 19:38:07.908629 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.908572 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7"} err="failed to get container status \"ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7\": rpc error: code = NotFound desc = could not find container \"ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7\": container with ID starting with ca9150d3bd7f4a8bce3bfff83f55aa80dd0fff1085d6c5a3d996a28910e54dd7 not found: ID does not exist" Jul 06 19:38:07.908629 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.908590 2571 scope.go:117] "RemoveContainer" containerID="69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4" Jul 06 19:38:07.908852 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:38:07.908835 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4\": container with ID starting with 69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4 not found: ID does not exist" containerID="69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4" Jul 06 19:38:07.908895 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:07.908857 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4"} err="failed to get container status \"69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4\": rpc error: code = NotFound desc = could not find container \"69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4\": container with ID starting with 69713e681bcc067e90368c3f1cee9e50c56c22fdac93600ac1270df57512ece4 not found: ID does not exist" Jul 06 19:38:08.006433 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:08.006375 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f6881-12cd-4ff4-849d-b2c2182d219d-extensions-socket-volume\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:38:08.006433 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:08.006395 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg2fn\" (UniqueName: \"kubernetes.io/projected/e58f6881-12cd-4ff4-849d-b2c2182d219d-kube-api-access-jg2fn\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:38:08.211838 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:08.211806 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c"] Jul 06 19:38:08.216812 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:08.216786 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prm5c"] Jul 06 19:38:09.461226 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:38:09.461196 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e58f6881-12cd-4ff4-849d-b2c2182d219d" path="/var/lib/kubelet/pods/e58f6881-12cd-4ff4-849d-b2c2182d219d/volumes" Jul 06 19:39:06.515678 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.515644 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh"] Jul 06 19:39:06.516149 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.516058 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e58f6881-12cd-4ff4-849d-b2c2182d219d" containerName="manager" Jul 06 19:39:06.516149 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.516069 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e58f6881-12cd-4ff4-849d-b2c2182d219d" containerName="manager" Jul 06 19:39:06.516149 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.516102 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e58f6881-12cd-4ff4-849d-b2c2182d219d" containerName="manager" Jul 06 19:39:06.516149 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.516107 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e58f6881-12cd-4ff4-849d-b2c2182d219d" containerName="manager" Jul 06 19:39:06.516304 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.516172 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e58f6881-12cd-4ff4-849d-b2c2182d219d" containerName="manager" Jul 06 19:39:06.516304 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.516183 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e58f6881-12cd-4ff4-849d-b2c2182d219d" containerName="manager" Jul 06 19:39:06.518223 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.518206 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:39:06.522048 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.522030 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p7bkg\"" Jul 06 19:39:06.532341 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.532320 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh"] Jul 06 19:39:06.627378 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.627350 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f796af-845b-47a2-9b37-e3186c4ddf66-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hpzdh\" (UID: \"45f796af-845b-47a2-9b37-e3186c4ddf66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:39:06.627510 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.627468 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kctzf\" (UniqueName: \"kubernetes.io/projected/45f796af-845b-47a2-9b37-e3186c4ddf66-kube-api-access-kctzf\") pod \"kuadrant-operator-controller-manager-86878c769f-hpzdh\" (UID: \"45f796af-845b-47a2-9b37-e3186c4ddf66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:39:06.728000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.727969 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kctzf\" (UniqueName: \"kubernetes.io/projected/45f796af-845b-47a2-9b37-e3186c4ddf66-kube-api-access-kctzf\") pod \"kuadrant-operator-controller-manager-86878c769f-hpzdh\" (UID: \"45f796af-845b-47a2-9b37-e3186c4ddf66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:39:06.728148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.728008 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f796af-845b-47a2-9b37-e3186c4ddf66-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hpzdh\" (UID: \"45f796af-845b-47a2-9b37-e3186c4ddf66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:39:06.728441 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.728423 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f796af-845b-47a2-9b37-e3186c4ddf66-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hpzdh\" (UID: \"45f796af-845b-47a2-9b37-e3186c4ddf66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:39:06.736736 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.736716 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kctzf\" (UniqueName: \"kubernetes.io/projected/45f796af-845b-47a2-9b37-e3186c4ddf66-kube-api-access-kctzf\") pod \"kuadrant-operator-controller-manager-86878c769f-hpzdh\" (UID: \"45f796af-845b-47a2-9b37-e3186c4ddf66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:39:06.828532 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.828458 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:39:06.977362 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:06.977334 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh"] Jul 06 19:39:06.979388 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:39:06.979357 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f796af_845b_47a2_9b37_e3186c4ddf66.slice/crio-f2b9e03cca12d4c4ef519d6795e64314403407280545ffdbf7fd90f05beafe06 WatchSource:0}: Error finding container f2b9e03cca12d4c4ef519d6795e64314403407280545ffdbf7fd90f05beafe06: Status 404 returned error can't find the container with id f2b9e03cca12d4c4ef519d6795e64314403407280545ffdbf7fd90f05beafe06 Jul 06 19:39:07.181407 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:07.181308 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" event={"ID":"45f796af-845b-47a2-9b37-e3186c4ddf66","Type":"ContainerStarted","Data":"09ad8b62cb21f555d9bbb984c98d35ff227112da49c9ca1a4c7d29269ed88cc6"} Jul 06 19:39:07.181407 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:07.181377 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" event={"ID":"45f796af-845b-47a2-9b37-e3186c4ddf66","Type":"ContainerStarted","Data":"f2b9e03cca12d4c4ef519d6795e64314403407280545ffdbf7fd90f05beafe06"} Jul 06 19:39:07.181631 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:07.181445 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:39:07.202019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:07.201966 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" podStartSLOduration=1.201952043 podStartE2EDuration="1.201952043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:07.201802095 +0000 UTC m=+1134.262149020" watchObservedRunningTime="2026-07-06 19:39:07.201952043 +0000 UTC m=+1134.262298969" Jul 06 19:39:18.190178 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:39:18.190073 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hpzdh" Jul 06 19:49:33.941198 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:33.941167 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l"] Jul 06 19:49:33.944763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:33.944747 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:33.947872 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:33.947852 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-78d446d5\"/\"kube-root-ca.crt\"" Jul 06 19:49:33.948066 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:33.948049 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-78d446d5\"/\"default-dockercfg-brd9f\"" Jul 06 19:49:33.948853 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:33.948833 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-78d446d5\"/\"e2e-lifecycle-model-78d446d5-kserve-self-signed-certs\"" Jul 06 19:49:33.948982 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:33.948869 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-78d446d5\"/\"openshift-service-ca.crt\"" Jul 06 19:49:33.963927 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:33.963905 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l"] Jul 06 19:49:34.067143 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.067102 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-home\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.067345 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.067155 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvr8j\" (UniqueName: \"kubernetes.io/projected/a3d40cb4-0394-4eb4-a78b-ac427a33851e-kube-api-access-dvr8j\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.067345 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.067227 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tmp-dir\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.067471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.067349 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-dshm\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.067471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.067404 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tls-certs\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.067471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.067434 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-model-cache\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.168450 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.168416 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-dshm\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.168618 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.168469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tls-certs\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.168618 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.168492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-model-cache\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.168618 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.168534 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-home\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.168618 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.168552 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvr8j\" (UniqueName: \"kubernetes.io/projected/a3d40cb4-0394-4eb4-a78b-ac427a33851e-kube-api-access-dvr8j\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.168618 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.168576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tmp-dir\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.168948 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.168920 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-model-cache\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.169069 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.168986 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-home\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.169069 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.169018 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tmp-dir\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.170945 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.170926 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-dshm\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.171043 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.171025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tls-certs\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.177399 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.177376 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvr8j\" (UniqueName: \"kubernetes.io/projected/a3d40cb4-0394-4eb4-a78b-ac427a33851e-kube-api-access-dvr8j\") pod \"e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.254479 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.254411 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:34.397292 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:34.397258 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l"] Jul 06 19:49:34.399919 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:49:34.399892 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3d40cb4_0394_4eb4_a78b_ac427a33851e.slice/crio-8d76445a9352c406fc4f39c703d7806517b67ebf424b0ef3944aed3cf130a514 WatchSource:0}: Error finding container 8d76445a9352c406fc4f39c703d7806517b67ebf424b0ef3944aed3cf130a514: Status 404 returned error can't find the container with id 8d76445a9352c406fc4f39c703d7806517b67ebf424b0ef3944aed3cf130a514 Jul 06 19:49:35.287563 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:35.287524 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" event={"ID":"a3d40cb4-0394-4eb4-a78b-ac427a33851e","Type":"ContainerStarted","Data":"ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf"} Jul 06 19:49:35.288019 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:35.287572 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" event={"ID":"a3d40cb4-0394-4eb4-a78b-ac427a33851e","Type":"ContainerStarted","Data":"8d76445a9352c406fc4f39c703d7806517b67ebf424b0ef3944aed3cf130a514"} Jul 06 19:49:35.334637 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:35.334587 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" podStartSLOduration=2.334571768 podStartE2EDuration="2.334571768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:49:35.333892636 +0000 UTC m=+1762.394239561" watchObservedRunningTime="2026-07-06 19:49:35.334571768 +0000 UTC m=+1762.394918693" Jul 06 19:49:44.254738 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:44.254699 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:44.255178 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:44.254831 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:44.271863 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:44.271836 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:49:44.346652 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:49:44.346628 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:50:17.631090 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:17.631050 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l"] Jul 06 19:50:32.652215 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:32.652162 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" podUID="a3d40cb4-0394-4eb4-a78b-ac427a33851e" containerName="main" containerID="cri-o://ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf" gracePeriod=45 Jul 06 19:50:32.908940 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:32.908888 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:50:33.023617 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.023590 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-dshm\") pod \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " Jul 06 19:50:33.023805 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.023647 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-home\") pod \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " Jul 06 19:50:33.023805 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.023672 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-model-cache\") pod \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " Jul 06 19:50:33.023805 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.023690 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tls-certs\") pod \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " Jul 06 19:50:33.023805 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.023775 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tmp-dir\") pod \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " Jul 06 19:50:33.023805 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.023794 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvr8j\" (UniqueName: \"kubernetes.io/projected/a3d40cb4-0394-4eb4-a78b-ac427a33851e-kube-api-access-dvr8j\") pod \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\" (UID: \"a3d40cb4-0394-4eb4-a78b-ac427a33851e\") " Jul 06 19:50:33.024089 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.023950 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-home" (OuterVolumeSpecName: "home") pod "a3d40cb4-0394-4eb4-a78b-ac427a33851e" (UID: "a3d40cb4-0394-4eb4-a78b-ac427a33851e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:33.024089 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.023985 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3d40cb4-0394-4eb4-a78b-ac427a33851e" (UID: "a3d40cb4-0394-4eb4-a78b-ac427a33851e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:33.024089 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.024031 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3d40cb4-0394-4eb4-a78b-ac427a33851e" (UID: "a3d40cb4-0394-4eb4-a78b-ac427a33851e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:33.024231 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.024163 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-home\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:50:33.024231 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.024179 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-model-cache\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:50:33.024231 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.024189 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tmp-dir\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:50:33.026017 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.025987 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-dshm" (OuterVolumeSpecName: "dshm") pod "a3d40cb4-0394-4eb4-a78b-ac427a33851e" (UID: "a3d40cb4-0394-4eb4-a78b-ac427a33851e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:33.026114 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.026041 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3d40cb4-0394-4eb4-a78b-ac427a33851e" (UID: "a3d40cb4-0394-4eb4-a78b-ac427a33851e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:50:33.026114 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.026076 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d40cb4-0394-4eb4-a78b-ac427a33851e-kube-api-access-dvr8j" (OuterVolumeSpecName: "kube-api-access-dvr8j") pod "a3d40cb4-0394-4eb4-a78b-ac427a33851e" (UID: "a3d40cb4-0394-4eb4-a78b-ac427a33851e"). InnerVolumeSpecName "kube-api-access-dvr8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:33.125244 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.125214 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d40cb4-0394-4eb4-a78b-ac427a33851e-tls-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:50:33.125244 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.125240 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvr8j\" (UniqueName: \"kubernetes.io/projected/a3d40cb4-0394-4eb4-a78b-ac427a33851e-kube-api-access-dvr8j\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:50:33.125405 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.125250 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d40cb4-0394-4eb4-a78b-ac427a33851e-dshm\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:50:33.583490 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.583455 2571 generic.go:356] "Generic (PLEG): container finished" podID="a3d40cb4-0394-4eb4-a78b-ac427a33851e" containerID="ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf" exitCode=0 Jul 06 19:50:33.583654 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.583514 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" Jul 06 19:50:33.583654 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.583540 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" event={"ID":"a3d40cb4-0394-4eb4-a78b-ac427a33851e","Type":"ContainerDied","Data":"ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf"} Jul 06 19:50:33.583654 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.583594 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l" event={"ID":"a3d40cb4-0394-4eb4-a78b-ac427a33851e","Type":"ContainerDied","Data":"8d76445a9352c406fc4f39c703d7806517b67ebf424b0ef3944aed3cf130a514"} Jul 06 19:50:33.583654 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.583612 2571 scope.go:117] "RemoveContainer" containerID="ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf" Jul 06 19:50:33.592547 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.592530 2571 scope.go:117] "RemoveContainer" containerID="ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf" Jul 06 19:50:33.592779 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:50:33.592762 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf\": container with ID starting with ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf not found: ID does not exist" containerID="ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf" Jul 06 19:50:33.592842 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.592790 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf"} err="failed to get container status \"ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf\": rpc error: code = NotFound desc = could not find container \"ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf\": container with ID starting with ca1fe5faeee9f6bfb191a0bd4153f15d2225957d5ceaefab700aaa7ce9ae80bf not found: ID does not exist" Jul 06 19:50:33.601480 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.601453 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l"] Jul 06 19:50:33.606288 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:33.606253 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-78d446d5/e2e-lifecycle-model-78d446d5-kserve-8645cb76d7-2st7l"] Jul 06 19:50:35.461586 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:50:35.461555 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d40cb4-0394-4eb4-a78b-ac427a33851e" path="/var/lib/kubelet/pods/a3d40cb4-0394-4eb4-a78b-ac427a33851e/volumes" Jul 06 19:52:09.119614 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.119575 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt"] Jul 06 19:52:09.120079 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.120055 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d40cb4-0394-4eb4-a78b-ac427a33851e" containerName="main" Jul 06 19:52:09.120079 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.120067 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d40cb4-0394-4eb4-a78b-ac427a33851e" containerName="main" Jul 06 19:52:09.120153 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.120146 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d40cb4-0394-4eb4-a78b-ac427a33851e" containerName="main" Jul 06 19:52:09.123387 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.123370 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.126219 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.126201 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:52:09.127375 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.127354 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:52:09.127486 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.127362 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-40d2cc-openshift-default-dockercfg-6vtrd\"" Jul 06 19:52:09.127709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.127692 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:52:09.143557 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.143524 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt"] Jul 06 19:52:09.189692 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.189665 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtgdr\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-kube-api-access-qtgdr\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.189804 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.189696 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-podinfo\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.189804 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.189716 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-token\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.189804 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.189768 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-socket\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.189912 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.189820 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-envoy\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.189912 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.189853 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istiod-ca-cert\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.189977 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.189958 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-data\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.190011 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.189989 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-credential-socket\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.190047 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.190007 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-certs\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.290864 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.290825 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtgdr\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-kube-api-access-qtgdr\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.290868 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-podinfo\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.290891 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-token\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.290915 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-socket\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.290939 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-envoy\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291015 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.290980 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istiod-ca-cert\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.291076 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-data\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.291128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-credential-socket\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.291153 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-certs\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291467 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.291444 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-socket\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291608 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.291586 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-credential-socket\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291671 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.291594 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-certs\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291671 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.291654 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-data\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.291948 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.291807 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istiod-ca-cert\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.293561 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.293536 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-envoy\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.293649 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.293636 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-podinfo\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.300540 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.300518 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtgdr\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-kube-api-access-qtgdr\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.301869 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.301840 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-token\") pod \"e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.434554 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.434466 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:09.582585 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.582559 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt"] Jul 06 19:52:09.584464 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:52:09.584429 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e5e071_6a5f_4b06_b65f_8c86fad0473d.slice/crio-2129b911e28a0844d04c835af2d15aa80d12173dd0c24041fb8d0c03095e7434 WatchSource:0}: Error finding container 2129b911e28a0844d04c835af2d15aa80d12173dd0c24041fb8d0c03095e7434: Status 404 returned error can't find the container with id 2129b911e28a0844d04c835af2d15aa80d12173dd0c24041fb8d0c03095e7434 Jul 06 19:52:09.586509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:09.586490 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:52:10.056086 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:10.056057 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" event={"ID":"01e5e071-6a5f-4b06-b65f-8c86fad0473d","Type":"ContainerStarted","Data":"2129b911e28a0844d04c835af2d15aa80d12173dd0c24041fb8d0c03095e7434"} Jul 06 19:52:12.382285 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:12.382227 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:52:12.382583 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:12.382322 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:52:12.382583 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:12.382353 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:52:13.075263 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:13.075228 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" event={"ID":"01e5e071-6a5f-4b06-b65f-8c86fad0473d","Type":"ContainerStarted","Data":"76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064"} Jul 06 19:52:13.106392 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:13.106343 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" podStartSLOduration=1.311086862 podStartE2EDuration="4.106327005s" totalImagesPullingTime="2.795240143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:13.103779235 +0000 UTC m=+1920.164126160" watchObservedRunningTime="2026-07-06 19:52:13.106327005 +0000 UTC m=+1920.166673931" Jul 06 19:52:13.435395 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:13.435307 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:13.439908 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:13.439881 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:14.080098 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:14.080071 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:14.080951 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:14.080934 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:28.152175 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:28.152143 2571 generic.go:356] "Generic (PLEG): container finished" podID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerID="76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064" exitCode=0 Jul 06 19:52:28.152664 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:28.152222 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" event={"ID":"01e5e071-6a5f-4b06-b65f-8c86fad0473d","Type":"ContainerDied","Data":"76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064"} Jul 06 19:52:28.152755 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:28.152687 2571 scope.go:117] "RemoveContainer" containerID="76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064" Jul 06 19:52:28.153737 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:28.153706 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:52:28.153836 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:28.153797 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:52:28.153900 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:28.153845 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:52:29.080626 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:29.080581 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:29.159796 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:29.159756 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" event={"ID":"01e5e071-6a5f-4b06-b65f-8c86fad0473d","Type":"ContainerStarted","Data":"e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb"} Jul 06 19:52:29.435253 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:29.435152 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:29.436383 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:29.436350 2571 patch_prober.go:28] interesting pod/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 06 19:52:29.436450 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:29.436425 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 06 19:52:30.435694 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:30.435655 2571 patch_prober.go:28] interesting pod/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 06 19:52:30.436093 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:30.435723 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 06 19:52:31.434902 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:31.434868 2571 patch_prober.go:28] interesting pod/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 06 19:52:31.435081 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:31.434927 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 06 19:52:32.435032 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:32.434996 2571 patch_prober.go:28] interesting pod/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 06 19:52:32.435442 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:32.435062 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 06 19:52:33.438709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:33.438677 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:33.439152 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:33.439003 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:33.439752 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:33.439730 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:52:50.016525 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.016487 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq"] Jul 06 19:52:50.020247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.020225 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.023739 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.023708 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-40d2cc\"/\"kube-root-ca.crt\"" Jul 06 19:52:50.023893 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.023716 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-40d2cc\"/\"default-dockercfg-t4sg4\"" Jul 06 19:52:50.023893 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.023715 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-40d2cc\"/\"openshift-service-ca.crt\"" Jul 06 19:52:50.023893 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.023824 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-40d2cc\"/\"test-model-40d2cc-kserve-self-signed-certs\"" Jul 06 19:52:50.032598 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.032577 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq"] Jul 06 19:52:50.079487 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.079456 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-home\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.079617 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.079494 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-dshm\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.079617 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.079534 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tmp-dir\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.079617 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.079558 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-model-cache\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.079617 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.079580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5kf4\" (UniqueName: \"kubernetes.io/projected/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-kube-api-access-v5kf4\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.079755 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.079664 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tls-certs\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.180737 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.180704 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-model-cache\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.180896 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.180745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5kf4\" (UniqueName: \"kubernetes.io/projected/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-kube-api-access-v5kf4\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.180896 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.180791 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tls-certs\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.180896 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.180826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-home\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.180896 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.180861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-dshm\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.181122 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.180916 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tmp-dir\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.181220 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.181176 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-model-cache\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.181316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.181216 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-home\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.181381 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.181359 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tmp-dir\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.183291 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.183255 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-dshm\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.183386 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.183338 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tls-certs\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.189333 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.189309 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5kf4\" (UniqueName: \"kubernetes.io/projected/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-kube-api-access-v5kf4\") pod \"test-model-40d2cc-kserve-644c7cfb5-vlnnq\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.331048 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.330966 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:52:50.469143 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:50.469117 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq"] Jul 06 19:52:50.471001 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:52:50.470975 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a01601_c7f4_4aa6_b52c_334a6a1ec917.slice/crio-dd822698151de63bf5dfcbe85c633585dba3e1e4371492bc3301fa408ac7bc8b WatchSource:0}: Error finding container dd822698151de63bf5dfcbe85c633585dba3e1e4371492bc3301fa408ac7bc8b: Status 404 returned error can't find the container with id dd822698151de63bf5dfcbe85c633585dba3e1e4371492bc3301fa408ac7bc8b Jul 06 19:52:51.277770 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:51.277740 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" event={"ID":"f6a01601-c7f4-4aa6-b52c-334a6a1ec917","Type":"ContainerStarted","Data":"c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb"} Jul 06 19:52:51.277770 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:51.277778 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" event={"ID":"f6a01601-c7f4-4aa6-b52c-334a6a1ec917","Type":"ContainerStarted","Data":"dd822698151de63bf5dfcbe85c633585dba3e1e4371492bc3301fa408ac7bc8b"} Jul 06 19:52:51.296707 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:52:51.296663 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" podStartSLOduration=2.29664488 podStartE2EDuration="2.29664488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:51.294995277 +0000 UTC m=+1958.355342205" watchObservedRunningTime="2026-07-06 19:52:51.29664488 +0000 UTC m=+1958.356991807" Jul 06 19:53:00.331588 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:00.331555 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:53:00.331588 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:00.331594 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:53:00.343893 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:00.343869 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:53:01.336471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:01.336442 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:53:13.083346 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.083244 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj"] Jul 06 19:53:13.087250 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.087230 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.091441 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.091411 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-reuse-bdf3a5-openshift-default-dockercfg-mxbz2\"" Jul 06 19:53:13.104724 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.104698 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-envoy\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.104852 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.104799 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-socket\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.104852 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.104838 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8bdfea-a0eb-4530-a739-759881070718-istio-podinfo\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.105000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.104903 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-istio-token\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.105071 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.105003 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8bdfea-a0eb-4530-a739-759881070718-istiod-ca-cert\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.105071 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.105063 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-data\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.105206 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.105174 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-certs\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.105306 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.105252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47p7q\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-kube-api-access-47p7q\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.105416 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.105389 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-credential-socket\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.108003 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.107982 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj"] Jul 06 19:53:13.206592 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.206561 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8bdfea-a0eb-4530-a739-759881070718-istiod-ca-cert\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.206776 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.206611 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-data\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.206776 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.206653 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-certs\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.206776 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.206692 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47p7q\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-kube-api-access-47p7q\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.206776 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.206761 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-credential-socket\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.207000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.206816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-envoy\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.207000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.206846 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-socket\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.207000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.206872 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8bdfea-a0eb-4530-a739-759881070718-istio-podinfo\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.207000 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.206901 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-istio-token\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.207257 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.207224 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-credential-socket\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.207470 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.207326 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-data\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.207555 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.207414 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-certs\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.207555 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.207439 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8bdfea-a0eb-4530-a739-759881070718-istiod-ca-cert\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.207555 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.207531 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-socket\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.209588 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.209562 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-envoy\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.209894 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.209875 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8bdfea-a0eb-4530-a739-759881070718-istio-podinfo\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.216536 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.216512 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47p7q\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-kube-api-access-47p7q\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.216869 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.216849 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-istio-token\") pod \"e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.403892 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.403811 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-reuse-bdf3a5-openshift-default-dockercfg-mxbz2\"" Jul 06 19:53:13.412071 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.412049 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:13.553515 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.553490 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj"] Jul 06 19:53:13.554784 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:53:13.554752 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8bdfea_a0eb_4530_a739_759881070718.slice/crio-10f0e38f4d190afaa099f4df689c60e36f4ad10d23e8110b34a75cfa3c98dacd WatchSource:0}: Error finding container 10f0e38f4d190afaa099f4df689c60e36f4ad10d23e8110b34a75cfa3c98dacd: Status 404 returned error can't find the container with id 10f0e38f4d190afaa099f4df689c60e36f4ad10d23e8110b34a75cfa3c98dacd Jul 06 19:53:13.556805 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.556773 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:53:13.556889 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.556839 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:53:13.556889 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:13.556869 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:53:14.392695 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:14.392659 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" event={"ID":"0e8bdfea-a0eb-4530-a739-759881070718","Type":"ContainerStarted","Data":"da2c593ae96c5911733633bd5ba9ce64d612f6914b5cd14199409edb6e45ff7a"} Jul 06 19:53:14.393082 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:14.392719 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" event={"ID":"0e8bdfea-a0eb-4530-a739-759881070718","Type":"ContainerStarted","Data":"10f0e38f4d190afaa099f4df689c60e36f4ad10d23e8110b34a75cfa3c98dacd"} Jul 06 19:53:14.412448 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:14.412423 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:14.416500 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:14.416452 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" podStartSLOduration=1.4164337200000001 podStartE2EDuration="1.41643372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:53:14.414111953 +0000 UTC m=+1981.474458879" watchObservedRunningTime="2026-07-06 19:53:14.41643372 +0000 UTC m=+1981.476780648" Jul 06 19:53:14.417556 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:14.417539 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:15.398327 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:15.398294 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:15.399298 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:15.399249 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:35.884634 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:35.884552 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2"] Jul 06 19:53:35.888313 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:35.888264 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:35.892723 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:35.892673 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-bdf3a5\"/\"openshift-service-ca.crt\"" Jul 06 19:53:35.893633 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:35.893584 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-bdf3a5\"/\"kube-root-ca.crt\"" Jul 06 19:53:35.893818 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:35.893797 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-bdf3a5\"/\"test-model-bdf3a5-kserve-self-signed-certs\"" Jul 06 19:53:35.893901 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:35.893806 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-bdf3a5\"/\"default-dockercfg-27sw7\"" Jul 06 19:53:35.905121 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:35.905096 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2"] Jul 06 19:53:36.028776 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.028732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-dshm\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.028955 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.028847 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-tmp-dir\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.028955 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.028925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wt85\" (UniqueName: \"kubernetes.io/projected/d03828f2-149b-4456-b37c-ffe7a939d1e2-kube-api-access-7wt85\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.029079 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.028956 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03828f2-149b-4456-b37c-ffe7a939d1e2-tls-certs\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.029079 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.028985 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-model-cache\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.029079 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.029006 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-home\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.130644 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.130591 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-dshm\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.130848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.130684 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-tmp-dir\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.130848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.130724 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wt85\" (UniqueName: \"kubernetes.io/projected/d03828f2-149b-4456-b37c-ffe7a939d1e2-kube-api-access-7wt85\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.130848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.130750 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03828f2-149b-4456-b37c-ffe7a939d1e2-tls-certs\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.131013 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.130881 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-model-cache\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.131013 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.130913 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-home\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.131337 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.131252 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-tmp-dir\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.131337 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.131302 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-home\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.131337 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.131334 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-model-cache\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.133573 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.133543 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-dshm\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.133709 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.133656 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03828f2-149b-4456-b37c-ffe7a939d1e2-tls-certs\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.140677 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.140607 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wt85\" (UniqueName: \"kubernetes.io/projected/d03828f2-149b-4456-b37c-ffe7a939d1e2-kube-api-access-7wt85\") pod \"test-model-bdf3a5-kserve-966c6bb75-wqxr2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.202688 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.202649 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:36.354684 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.354655 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2"] Jul 06 19:53:36.360166 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:53:36.356422 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd03828f2_149b_4456_b37c_ffe7a939d1e2.slice/crio-ae8b14a93b2853a3739a1de62e28d71f224abc28724baaad5f917c4a31db9989 WatchSource:0}: Error finding container ae8b14a93b2853a3739a1de62e28d71f224abc28724baaad5f917c4a31db9989: Status 404 returned error can't find the container with id ae8b14a93b2853a3739a1de62e28d71f224abc28724baaad5f917c4a31db9989 Jul 06 19:53:36.515959 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.515921 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" event={"ID":"d03828f2-149b-4456-b37c-ffe7a939d1e2","Type":"ContainerStarted","Data":"b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c"} Jul 06 19:53:36.516158 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.515970 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" event={"ID":"d03828f2-149b-4456-b37c-ffe7a939d1e2","Type":"ContainerStarted","Data":"ae8b14a93b2853a3739a1de62e28d71f224abc28724baaad5f917c4a31db9989"} Jul 06 19:53:36.536173 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:36.536120 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" podStartSLOduration=1.5361046219999999 podStartE2EDuration="1.536104622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:53:36.53298245 +0000 UTC m=+2003.593329358" watchObservedRunningTime="2026-07-06 19:53:36.536104622 +0000 UTC m=+2003.596451548" Jul 06 19:53:39.531659 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:39.531628 2571 generic.go:356] "Generic (PLEG): container finished" podID="0e8bdfea-a0eb-4530-a739-759881070718" containerID="da2c593ae96c5911733633bd5ba9ce64d612f6914b5cd14199409edb6e45ff7a" exitCode=0 Jul 06 19:53:39.532076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:39.531695 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" event={"ID":"0e8bdfea-a0eb-4530-a739-759881070718","Type":"ContainerDied","Data":"da2c593ae96c5911733633bd5ba9ce64d612f6914b5cd14199409edb6e45ff7a"} Jul 06 19:53:39.532076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:39.532070 2571 scope.go:117] "RemoveContainer" containerID="da2c593ae96c5911733633bd5ba9ce64d612f6914b5cd14199409edb6e45ff7a" Jul 06 19:53:39.533071 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:39.533046 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:53:39.533125 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:39.533111 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:53:39.533163 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:39.533142 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:53:40.538715 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:40.538676 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" event={"ID":"0e8bdfea-a0eb-4530-a739-759881070718","Type":"ContainerStarted","Data":"c01323dfbe9048fbd4f8f9e358b7e9ef43428d1579665cfa1e34c5ecbba04ce0"} Jul 06 19:53:41.412288 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:41.412244 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:41.413192 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:41.413166 2571 patch_prober.go:28] interesting pod/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 06 19:53:41.413306 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:41.413219 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 06 19:53:42.412803 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:42.412766 2571 patch_prober.go:28] interesting pod/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 06 19:53:42.413335 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:42.412828 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 06 19:53:43.412579 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:43.412536 2571 patch_prober.go:28] interesting pod/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 06 19:53:43.412794 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:43.412592 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 06 19:53:44.413135 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:44.413100 2571 patch_prober.go:28] interesting pod/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 06 19:53:44.413552 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:44.413162 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 06 19:53:45.398345 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:45.398312 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:45.417247 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:45.417224 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:45.567091 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:45.567060 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:53:46.203700 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:46.203667 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:46.203874 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:46.203725 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:46.216161 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:46.216135 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:46.581834 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:46.581800 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:53:52.950939 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:53:52.950904 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2"] Jul 06 19:54:07.972113 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:07.972064 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" podUID="d03828f2-149b-4456-b37c-ffe7a939d1e2" containerName="main" containerID="cri-o://b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c" gracePeriod=45 Jul 06 19:54:08.221866 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.221839 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:54:08.343781 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.343742 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03828f2-149b-4456-b37c-ffe7a939d1e2-tls-certs\") pod \"d03828f2-149b-4456-b37c-ffe7a939d1e2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " Jul 06 19:54:08.343976 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.343804 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-tmp-dir\") pod \"d03828f2-149b-4456-b37c-ffe7a939d1e2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " Jul 06 19:54:08.343976 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.343835 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-home\") pod \"d03828f2-149b-4456-b37c-ffe7a939d1e2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " Jul 06 19:54:08.343976 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.343889 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-model-cache\") pod \"d03828f2-149b-4456-b37c-ffe7a939d1e2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " Jul 06 19:54:08.343976 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.343914 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wt85\" (UniqueName: \"kubernetes.io/projected/d03828f2-149b-4456-b37c-ffe7a939d1e2-kube-api-access-7wt85\") pod \"d03828f2-149b-4456-b37c-ffe7a939d1e2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " Jul 06 19:54:08.343976 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.343959 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-dshm\") pod \"d03828f2-149b-4456-b37c-ffe7a939d1e2\" (UID: \"d03828f2-149b-4456-b37c-ffe7a939d1e2\") " Jul 06 19:54:08.344231 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.344078 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d03828f2-149b-4456-b37c-ffe7a939d1e2" (UID: "d03828f2-149b-4456-b37c-ffe7a939d1e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:08.344231 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.344140 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "d03828f2-149b-4456-b37c-ffe7a939d1e2" (UID: "d03828f2-149b-4456-b37c-ffe7a939d1e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:08.344231 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.344124 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-home" (OuterVolumeSpecName: "home") pod "d03828f2-149b-4456-b37c-ffe7a939d1e2" (UID: "d03828f2-149b-4456-b37c-ffe7a939d1e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:08.344422 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.344349 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-home\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:08.344422 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.344361 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-model-cache\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:08.344422 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.344371 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-tmp-dir\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:08.346148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.346113 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03828f2-149b-4456-b37c-ffe7a939d1e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d03828f2-149b-4456-b37c-ffe7a939d1e2" (UID: "d03828f2-149b-4456-b37c-ffe7a939d1e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:54:08.346148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.346119 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-dshm" (OuterVolumeSpecName: "dshm") pod "d03828f2-149b-4456-b37c-ffe7a939d1e2" (UID: "d03828f2-149b-4456-b37c-ffe7a939d1e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:08.346148 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.346126 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03828f2-149b-4456-b37c-ffe7a939d1e2-kube-api-access-7wt85" (OuterVolumeSpecName: "kube-api-access-7wt85") pod "d03828f2-149b-4456-b37c-ffe7a939d1e2" (UID: "d03828f2-149b-4456-b37c-ffe7a939d1e2"). InnerVolumeSpecName "kube-api-access-7wt85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:08.445020 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.444977 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wt85\" (UniqueName: \"kubernetes.io/projected/d03828f2-149b-4456-b37c-ffe7a939d1e2-kube-api-access-7wt85\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:08.445020 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.445016 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d03828f2-149b-4456-b37c-ffe7a939d1e2-dshm\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:08.445020 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.445026 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d03828f2-149b-4456-b37c-ffe7a939d1e2-tls-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:08.688804 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.688712 2571 generic.go:356] "Generic (PLEG): container finished" podID="d03828f2-149b-4456-b37c-ffe7a939d1e2" containerID="b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c" exitCode=0 Jul 06 19:54:08.688804 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.688755 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" Jul 06 19:54:08.688991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.688797 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" event={"ID":"d03828f2-149b-4456-b37c-ffe7a939d1e2","Type":"ContainerDied","Data":"b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c"} Jul 06 19:54:08.688991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.688844 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2" event={"ID":"d03828f2-149b-4456-b37c-ffe7a939d1e2","Type":"ContainerDied","Data":"ae8b14a93b2853a3739a1de62e28d71f224abc28724baaad5f917c4a31db9989"} Jul 06 19:54:08.688991 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.688861 2571 scope.go:117] "RemoveContainer" containerID="b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c" Jul 06 19:54:08.698224 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.698205 2571 scope.go:117] "RemoveContainer" containerID="b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c" Jul 06 19:54:08.698509 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:54:08.698488 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c\": container with ID starting with b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c not found: ID does not exist" containerID="b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c" Jul 06 19:54:08.698567 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.698516 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c"} err="failed to get container status \"b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c\": rpc error: code = NotFound desc = could not find container \"b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c\": container with ID starting with b5d0c95da6a474f0fadaf53818a45ee26ae33872491ae4d1120404e42a7e7e3c not found: ID does not exist" Jul 06 19:54:08.716464 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.716443 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2"] Jul 06 19:54:08.722649 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:08.722629 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-reuse-bdf3a5/test-model-bdf3a5-kserve-966c6bb75-wqxr2"] Jul 06 19:54:09.462794 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:09.462762 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d03828f2-149b-4456-b37c-ffe7a939d1e2" path="/var/lib/kubelet/pods/d03828f2-149b-4456-b37c-ffe7a939d1e2/volumes" Jul 06 19:54:36.735229 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:36.735147 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj"] Jul 06 19:54:36.735690 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:36.735409 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" containerID="cri-o://c01323dfbe9048fbd4f8f9e358b7e9ef43428d1579665cfa1e34c5ecbba04ce0" gracePeriod=30 Jul 06 19:54:38.964342 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:38.964300 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq"] Jul 06 19:54:41.858485 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:41.858453 2571 generic.go:356] "Generic (PLEG): container finished" podID="0e8bdfea-a0eb-4530-a739-759881070718" containerID="c01323dfbe9048fbd4f8f9e358b7e9ef43428d1579665cfa1e34c5ecbba04ce0" exitCode=0 Jul 06 19:54:41.858880 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:41.858483 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" event={"ID":"0e8bdfea-a0eb-4530-a739-759881070718","Type":"ContainerDied","Data":"c01323dfbe9048fbd4f8f9e358b7e9ef43428d1579665cfa1e34c5ecbba04ce0"} Jul 06 19:54:41.858880 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:41.858563 2571 scope.go:117] "RemoveContainer" containerID="da2c593ae96c5911733633bd5ba9ce64d612f6914b5cd14199409edb6e45ff7a" Jul 06 19:54:42.001751 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.001691 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:54:42.063417 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063384 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-socket\") pod \"0e8bdfea-a0eb-4530-a739-759881070718\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " Jul 06 19:54:42.063642 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063433 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8bdfea-a0eb-4530-a739-759881070718-istio-podinfo\") pod \"0e8bdfea-a0eb-4530-a739-759881070718\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " Jul 06 19:54:42.063642 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063480 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47p7q\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-kube-api-access-47p7q\") pod \"0e8bdfea-a0eb-4530-a739-759881070718\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " Jul 06 19:54:42.063642 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063520 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-data\") pod \"0e8bdfea-a0eb-4530-a739-759881070718\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " Jul 06 19:54:42.063642 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063579 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8bdfea-a0eb-4530-a739-759881070718-istiod-ca-cert\") pod \"0e8bdfea-a0eb-4530-a739-759881070718\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063657 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e8bdfea-a0eb-4530-a739-759881070718" (UID: "0e8bdfea-a0eb-4530-a739-759881070718"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063676 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-credential-socket\") pod \"0e8bdfea-a0eb-4530-a739-759881070718\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063704 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-certs\") pod \"0e8bdfea-a0eb-4530-a739-759881070718\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063732 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-envoy\") pod \"0e8bdfea-a0eb-4530-a739-759881070718\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.063772 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-istio-token\") pod \"0e8bdfea-a0eb-4530-a739-759881070718\" (UID: \"0e8bdfea-a0eb-4530-a739-759881070718\") " Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.064012 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e8bdfea-a0eb-4530-a739-759881070718" (UID: "0e8bdfea-a0eb-4530-a739-759881070718"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.064167 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-socket\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.064189 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-credential-socket\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.064222 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e8bdfea-a0eb-4530-a739-759881070718" (UID: "0e8bdfea-a0eb-4530-a739-759881070718"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:42.064338 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.064240 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8bdfea-a0eb-4530-a739-759881070718-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e8bdfea-a0eb-4530-a739-759881070718" (UID: "0e8bdfea-a0eb-4530-a739-759881070718"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:54:42.065132 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.064652 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e8bdfea-a0eb-4530-a739-759881070718" (UID: "0e8bdfea-a0eb-4530-a739-759881070718"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:42.066203 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.066180 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e8bdfea-a0eb-4530-a739-759881070718-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e8bdfea-a0eb-4530-a739-759881070718" (UID: "0e8bdfea-a0eb-4530-a739-759881070718"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:54:42.066696 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.066667 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e8bdfea-a0eb-4530-a739-759881070718" (UID: "0e8bdfea-a0eb-4530-a739-759881070718"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:42.066950 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.066920 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-kube-api-access-47p7q" (OuterVolumeSpecName: "kube-api-access-47p7q") pod "0e8bdfea-a0eb-4530-a739-759881070718" (UID: "0e8bdfea-a0eb-4530-a739-759881070718"). InnerVolumeSpecName "kube-api-access-47p7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:42.067849 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.067830 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e8bdfea-a0eb-4530-a739-759881070718" (UID: "0e8bdfea-a0eb-4530-a739-759881070718"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:42.165595 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.165552 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8bdfea-a0eb-4530-a739-759881070718-istio-podinfo\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.165595 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.165589 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47p7q\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-kube-api-access-47p7q\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.165595 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.165600 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-data\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.165595 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.165607 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8bdfea-a0eb-4530-a739-759881070718-istiod-ca-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.165947 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.165616 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-workload-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.165947 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.165623 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8bdfea-a0eb-4530-a739-759881070718-istio-envoy\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.165947 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.165630 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8bdfea-a0eb-4530-a739-759881070718-istio-token\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.869444 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.869395 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" event={"ID":"0e8bdfea-a0eb-4530-a739-759881070718","Type":"ContainerDied","Data":"10f0e38f4d190afaa099f4df689c60e36f4ad10d23e8110b34a75cfa3c98dacd"} Jul 06 19:54:42.870030 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.869406 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj" Jul 06 19:54:42.870030 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.869514 2571 scope.go:117] "RemoveContainer" containerID="c01323dfbe9048fbd4f8f9e358b7e9ef43428d1579665cfa1e34c5ecbba04ce0" Jul 06 19:54:42.891795 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.891765 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj"] Jul 06 19:54:42.895945 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:42.895923 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-reuse-bdf3a5-openshift-default-78f6848497-p48pj"] Jul 06 19:54:43.463119 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:43.463078 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e8bdfea-a0eb-4530-a739-759881070718" path="/var/lib/kubelet/pods/0e8bdfea-a0eb-4530-a739-759881070718/volumes" Jul 06 19:54:53.985102 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:53.985044 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" podUID="f6a01601-c7f4-4aa6-b52c-334a6a1ec917" containerName="main" containerID="cri-o://c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb" gracePeriod=45 Jul 06 19:54:54.235842 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.235786 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:54:54.381423 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.381390 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5kf4\" (UniqueName: \"kubernetes.io/projected/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-kube-api-access-v5kf4\") pod \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " Jul 06 19:54:54.381604 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.381495 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-home\") pod \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " Jul 06 19:54:54.381604 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.381556 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tmp-dir\") pod \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " Jul 06 19:54:54.381726 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.381604 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-model-cache\") pod \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " Jul 06 19:54:54.381726 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.381668 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tls-certs\") pod \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " Jul 06 19:54:54.381726 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.381701 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-dshm\") pod \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\" (UID: \"f6a01601-c7f4-4aa6-b52c-334a6a1ec917\") " Jul 06 19:54:54.381880 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.381770 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6a01601-c7f4-4aa6-b52c-334a6a1ec917" (UID: "f6a01601-c7f4-4aa6-b52c-334a6a1ec917"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:54.381880 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.381780 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-home" (OuterVolumeSpecName: "home") pod "f6a01601-c7f4-4aa6-b52c-334a6a1ec917" (UID: "f6a01601-c7f4-4aa6-b52c-334a6a1ec917"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:54.381880 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.381861 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6a01601-c7f4-4aa6-b52c-334a6a1ec917" (UID: "f6a01601-c7f4-4aa6-b52c-334a6a1ec917"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:54.382051 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.382029 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-home\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:54.382051 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.382052 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tmp-dir\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:54.382237 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.382063 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-model-cache\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:54.383938 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.383910 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-kube-api-access-v5kf4" (OuterVolumeSpecName: "kube-api-access-v5kf4") pod "f6a01601-c7f4-4aa6-b52c-334a6a1ec917" (UID: "f6a01601-c7f4-4aa6-b52c-334a6a1ec917"). InnerVolumeSpecName "kube-api-access-v5kf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:54.384028 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.383975 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6a01601-c7f4-4aa6-b52c-334a6a1ec917" (UID: "f6a01601-c7f4-4aa6-b52c-334a6a1ec917"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:54:54.384028 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.384016 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-dshm" (OuterVolumeSpecName: "dshm") pod "f6a01601-c7f4-4aa6-b52c-334a6a1ec917" (UID: "f6a01601-c7f4-4aa6-b52c-334a6a1ec917"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:54.483443 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.483403 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-tls-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:54.483443 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.483438 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-dshm\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:54.483653 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.483452 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5kf4\" (UniqueName: \"kubernetes.io/projected/f6a01601-c7f4-4aa6-b52c-334a6a1ec917-kube-api-access-v5kf4\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:54:54.933839 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.933805 2571 generic.go:356] "Generic (PLEG): container finished" podID="f6a01601-c7f4-4aa6-b52c-334a6a1ec917" containerID="c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb" exitCode=0 Jul 06 19:54:54.934069 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.933851 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" Jul 06 19:54:54.934069 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.933878 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" event={"ID":"f6a01601-c7f4-4aa6-b52c-334a6a1ec917","Type":"ContainerDied","Data":"c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb"} Jul 06 19:54:54.934069 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.933916 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq" event={"ID":"f6a01601-c7f4-4aa6-b52c-334a6a1ec917","Type":"ContainerDied","Data":"dd822698151de63bf5dfcbe85c633585dba3e1e4371492bc3301fa408ac7bc8b"} Jul 06 19:54:54.934069 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.933937 2571 scope.go:117] "RemoveContainer" containerID="c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb" Jul 06 19:54:54.943189 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.943170 2571 scope.go:117] "RemoveContainer" containerID="c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb" Jul 06 19:54:54.943473 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:54:54.943452 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb\": container with ID starting with c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb not found: ID does not exist" containerID="c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb" Jul 06 19:54:54.943574 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.943480 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb"} err="failed to get container status \"c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb\": rpc error: code = NotFound desc = could not find container \"c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb\": container with ID starting with c4e9b28c8e015250907834cffef24994f9c6983359cfdf75b9bb3026e4f52ffb not found: ID does not exist" Jul 06 19:54:54.955401 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.955372 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq"] Jul 06 19:54:54.957785 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:54.957765 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-40d2cc/test-model-40d2cc-kserve-644c7cfb5-vlnnq"] Jul 06 19:54:55.463768 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:54:55.463729 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a01601-c7f4-4aa6-b52c-334a6a1ec917" path="/var/lib/kubelet/pods/f6a01601-c7f4-4aa6-b52c-334a6a1ec917/volumes" Jul 06 19:55:29.522342 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:29.522307 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt"] Jul 06 19:55:29.522843 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:29.522538 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" containerID="cri-o://e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb" gracePeriod=30 Jul 06 19:55:33.441920 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:33.441880 2571 patch_prober.go:28] interesting pod/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:55:33.442408 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:33.441962 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:55:34.785340 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.785257 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:55:34.963975 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.963930 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-socket\") pod \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " Jul 06 19:55:34.964174 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964005 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtgdr\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-kube-api-access-qtgdr\") pod \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " Jul 06 19:55:34.964174 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964051 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istiod-ca-cert\") pod \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " Jul 06 19:55:34.964174 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964072 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-token\") pod \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " Jul 06 19:55:34.964174 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964097 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-podinfo\") pod \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " Jul 06 19:55:34.964174 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964133 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-certs\") pod \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " Jul 06 19:55:34.964482 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964179 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-credential-socket\") pod \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " Jul 06 19:55:34.964482 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964216 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-data\") pod \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " Jul 06 19:55:34.964482 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964240 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-envoy\") pod \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\" (UID: \"01e5e071-6a5f-4b06-b65f-8c86fad0473d\") " Jul 06 19:55:34.964482 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964256 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "01e5e071-6a5f-4b06-b65f-8c86fad0473d" (UID: "01e5e071-6a5f-4b06-b65f-8c86fad0473d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:34.964690 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964519 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "01e5e071-6a5f-4b06-b65f-8c86fad0473d" (UID: "01e5e071-6a5f-4b06-b65f-8c86fad0473d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:34.964690 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964603 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "01e5e071-6a5f-4b06-b65f-8c86fad0473d" (UID: "01e5e071-6a5f-4b06-b65f-8c86fad0473d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:55:34.964690 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964670 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-credential-socket\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:55:34.964843 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.964689 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-socket\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:55:34.965126 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.965057 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "01e5e071-6a5f-4b06-b65f-8c86fad0473d" (UID: "01e5e071-6a5f-4b06-b65f-8c86fad0473d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:34.965215 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.965178 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-data" (OuterVolumeSpecName: "istio-data") pod "01e5e071-6a5f-4b06-b65f-8c86fad0473d" (UID: "01e5e071-6a5f-4b06-b65f-8c86fad0473d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:34.967474 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.967444 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "01e5e071-6a5f-4b06-b65f-8c86fad0473d" (UID: "01e5e071-6a5f-4b06-b65f-8c86fad0473d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:34.967580 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.967526 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "01e5e071-6a5f-4b06-b65f-8c86fad0473d" (UID: "01e5e071-6a5f-4b06-b65f-8c86fad0473d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:55:34.967630 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.967585 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-token" (OuterVolumeSpecName: "istio-token") pod "01e5e071-6a5f-4b06-b65f-8c86fad0473d" (UID: "01e5e071-6a5f-4b06-b65f-8c86fad0473d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:55:34.967630 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:34.967585 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-kube-api-access-qtgdr" (OuterVolumeSpecName: "kube-api-access-qtgdr") pod "01e5e071-6a5f-4b06-b65f-8c86fad0473d" (UID: "01e5e071-6a5f-4b06-b65f-8c86fad0473d"). InnerVolumeSpecName "kube-api-access-qtgdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:55:35.066179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.066095 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtgdr\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-kube-api-access-qtgdr\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:55:35.066179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.066128 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istiod-ca-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:55:35.066179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.066138 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-token\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:55:35.066179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.066146 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-podinfo\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:55:35.066179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.066154 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-workload-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:55:35.066179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.066164 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-data\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:55:35.066179 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.066174 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e5e071-6a5f-4b06-b65f-8c86fad0473d-istio-envoy\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:55:35.147008 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.146976 2571 generic.go:356] "Generic (PLEG): container finished" podID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerID="e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb" exitCode=0 Jul 06 19:55:35.147160 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.147092 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" Jul 06 19:55:35.147160 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.147118 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" event={"ID":"01e5e071-6a5f-4b06-b65f-8c86fad0473d","Type":"ContainerDied","Data":"e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb"} Jul 06 19:55:35.147296 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.147178 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt" event={"ID":"01e5e071-6a5f-4b06-b65f-8c86fad0473d","Type":"ContainerDied","Data":"2129b911e28a0844d04c835af2d15aa80d12173dd0c24041fb8d0c03095e7434"} Jul 06 19:55:35.147296 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.147209 2571 scope.go:117] "RemoveContainer" containerID="e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb" Jul 06 19:55:35.157718 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.157688 2571 scope.go:117] "RemoveContainer" containerID="76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064" Jul 06 19:55:35.168502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.168478 2571 scope.go:117] "RemoveContainer" containerID="e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb" Jul 06 19:55:35.168827 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:55:35.168795 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb\": container with ID starting with e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb not found: ID does not exist" containerID="e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb" Jul 06 19:55:35.168915 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.168839 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb"} err="failed to get container status \"e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb\": rpc error: code = NotFound desc = could not find container \"e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb\": container with ID starting with e0f4180d0240edeee162314de57140d909eae66ddf220b917b5697de7b206ffb not found: ID does not exist" Jul 06 19:55:35.168915 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.168865 2571 scope.go:117] "RemoveContainer" containerID="76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064" Jul 06 19:55:35.169179 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:55:35.169158 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064\": container with ID starting with 76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064 not found: ID does not exist" containerID="76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064" Jul 06 19:55:35.169259 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.169192 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064"} err="failed to get container status \"76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064\": rpc error: code = NotFound desc = could not find container \"76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064\": container with ID starting with 76f68f74465439917fd0dccdaf7d5a1a48af06f9ad35475d666c1fc43e8fd064 not found: ID does not exist" Jul 06 19:55:35.178084 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.178057 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt"] Jul 06 19:55:35.199459 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.199430 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-40d2cc-openshift-default-65dfd8dcc9-t46tt"] Jul 06 19:55:35.465411 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:35.465311 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" path="/var/lib/kubelet/pods/01e5e071-6a5f-4b06-b65f-8c86fad0473d/volumes" Jul 06 19:55:39.120471 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.120441 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k"] Jul 06 19:55:39.121010 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.120993 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a01601-c7f4-4aa6-b52c-334a6a1ec917" containerName="main" Jul 06 19:55:39.121056 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121012 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a01601-c7f4-4aa6-b52c-334a6a1ec917" containerName="main" Jul 06 19:55:39.121056 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121031 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" Jul 06 19:55:39.121056 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121052 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" Jul 06 19:55:39.121150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121061 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" Jul 06 19:55:39.121150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121067 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" Jul 06 19:55:39.121150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121081 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" Jul 06 19:55:39.121150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121087 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" Jul 06 19:55:39.121150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121095 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03828f2-149b-4456-b37c-ffe7a939d1e2" containerName="main" Jul 06 19:55:39.121150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121100 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03828f2-149b-4456-b37c-ffe7a939d1e2" containerName="main" Jul 06 19:55:39.121150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121105 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" Jul 06 19:55:39.121150 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121110 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" Jul 06 19:55:39.121412 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121172 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d03828f2-149b-4456-b37c-ffe7a939d1e2" containerName="main" Jul 06 19:55:39.121412 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121181 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a01601-c7f4-4aa6-b52c-334a6a1ec917" containerName="main" Jul 06 19:55:39.121412 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121188 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" Jul 06 19:55:39.121412 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121194 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" Jul 06 19:55:39.121412 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121199 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e8bdfea-a0eb-4530-a739-759881070718" containerName="istio-proxy" Jul 06 19:55:39.121412 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.121205 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e5e071-6a5f-4b06-b65f-8c86fad0473d" containerName="istio-proxy" Jul 06 19:55:39.125798 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.125774 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.129397 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.129366 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:55:39.129534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.129432 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bab069-openshift-default-dockercfg-2rkhr\"" Jul 06 19:55:39.129534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.129457 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:55:39.129534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.129511 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:55:39.137057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.137032 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k"] Jul 06 19:55:39.309658 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.309620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-credential-socket\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.309658 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.309655 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zcwb\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-kube-api-access-7zcwb\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.309905 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.309689 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-envoy\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.309905 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.309705 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-token\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.309905 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.309732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-socket\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.309905 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.309753 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-podinfo\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.309905 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.309769 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-data\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.309905 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.309883 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istiod-ca-cert\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.310138 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.309938 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-certs\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411312 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411195 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-envoy\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411312 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411231 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-token\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411312 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411259 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-socket\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411650 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-podinfo\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411650 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411334 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-data\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411650 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411384 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istiod-ca-cert\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411650 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411438 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-certs\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411650 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-credential-socket\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411650 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411531 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zcwb\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-kube-api-access-7zcwb\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411986 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411773 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-socket\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411986 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411804 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-data\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411986 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411889 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-certs\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.411986 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.411950 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-credential-socket\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.412376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.412136 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istiod-ca-cert\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.413820 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.413787 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-envoy\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.413937 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.413922 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-podinfo\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.419512 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.419488 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-token\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.419751 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.419729 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zcwb\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-kube-api-access-7zcwb\") pod \"e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.436545 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.436520 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:39.588939 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:55:39.588893 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3dccaf_4e3f_4974_b21c_1bd5925f2c4b.slice/crio-a7cc321ef11d4392435e2633a5058ddfd8448b057f84a201efa1caa0cd6ab1f2 WatchSource:0}: Error finding container a7cc321ef11d4392435e2633a5058ddfd8448b057f84a201efa1caa0cd6ab1f2: Status 404 returned error can't find the container with id a7cc321ef11d4392435e2633a5058ddfd8448b057f84a201efa1caa0cd6ab1f2 Jul 06 19:55:39.599316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.593444 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:55:39.599316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.593609 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:55:39.599316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.593850 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:55:39.599316 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:39.597181 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k"] Jul 06 19:55:40.175624 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:40.175586 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" event={"ID":"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b","Type":"ContainerStarted","Data":"04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15"} Jul 06 19:55:40.176024 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:40.175639 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" event={"ID":"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b","Type":"ContainerStarted","Data":"a7cc321ef11d4392435e2633a5058ddfd8448b057f84a201efa1caa0cd6ab1f2"} Jul 06 19:55:40.200313 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:40.200246 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" podStartSLOduration=1.200227408 podStartE2EDuration="1.200227408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:55:40.197489519 +0000 UTC m=+2127.257836448" watchObservedRunningTime="2026-07-06 19:55:40.200227408 +0000 UTC m=+2127.260574334" Jul 06 19:55:40.437111 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:40.437019 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:40.442326 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:40.442297 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:41.180857 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:41.180824 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:41.181705 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:41.181683 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:57.222462 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:57.222415 2571 patch_prober.go:28] interesting pod/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 19:55:57.222911 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:57.222492 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:55:58.267792 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:58.267754 2571 generic.go:356] "Generic (PLEG): container finished" podID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerID="04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15" exitCode=0 Jul 06 19:55:58.268294 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:58.267828 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" event={"ID":"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b","Type":"ContainerDied","Data":"04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15"} Jul 06 19:55:58.268294 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:58.268232 2571 scope.go:117] "RemoveContainer" containerID="04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15" Jul 06 19:55:58.269413 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:58.269384 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:55:58.269509 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:58.269491 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:55:58.269553 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:58.269539 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:55:59.278502 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:59.278458 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" event={"ID":"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b","Type":"ContainerStarted","Data":"e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3"} Jul 06 19:55:59.436770 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:59.436728 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:55:59.436957 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:59.436934 2571 patch_prober.go:28] interesting pod/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 06 19:55:59.437010 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:55:59.436986 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 06 19:56:00.437385 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:00.437347 2571 patch_prober.go:28] interesting pod/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 06 19:56:00.437812 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:00.437400 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 06 19:56:01.437211 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:01.437174 2571 patch_prober.go:28] interesting pod/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 06 19:56:01.437455 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:01.437239 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 06 19:56:02.437031 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:02.436992 2571 patch_prober.go:28] interesting pod/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 06 19:56:02.437214 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:02.437051 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 06 19:56:03.440874 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:03.440844 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:56:03.441315 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:03.441178 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:56:03.441941 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:03.441919 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:56:09.843510 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:09.843469 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k"] Jul 06 19:56:09.843900 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:09.843669 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" containerID="cri-o://e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3" gracePeriod=30 Jul 06 19:56:15.099628 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.099598 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:56:15.260202 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.260093 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-credential-socket\") pod \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " Jul 06 19:56:15.260202 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.260144 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zcwb\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-kube-api-access-7zcwb\") pod \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " Jul 06 19:56:15.260202 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.260185 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-token\") pod \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " Jul 06 19:56:15.260529 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.260240 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istiod-ca-cert\") pod \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " Jul 06 19:56:15.260529 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.260306 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-envoy\") pod \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " Jul 06 19:56:15.260529 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.260342 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-data\") pod \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " Jul 06 19:56:15.260529 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.260372 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-socket\") pod \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " Jul 06 19:56:15.260744 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.260723 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" (UID: "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:56:15.261057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.261022 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" (UID: "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:15.261057 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.261035 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-podinfo\") pod \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " Jul 06 19:56:15.261221 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.261082 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-certs\") pod \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\" (UID: \"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b\") " Jul 06 19:56:15.261746 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.261408 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" (UID: "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:15.261746 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.261515 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-credential-socket\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:56:15.261746 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.261573 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" (UID: "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:15.261746 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.261700 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" (UID: "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:15.262012 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.261537 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istiod-ca-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:56:15.262012 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.261844 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-data\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:56:15.263613 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.263583 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" (UID: "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:15.264076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.264047 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-kube-api-access-7zcwb" (OuterVolumeSpecName: "kube-api-access-7zcwb") pod "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" (UID: "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b"). InnerVolumeSpecName "kube-api-access-7zcwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:56:15.264076 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.264059 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" (UID: "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:56:15.265284 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.265245 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" (UID: "ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:56:15.363230 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.363188 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-token\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:56:15.363230 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.363229 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-envoy\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:56:15.363490 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.363243 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-socket\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:56:15.363490 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.363257 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-istio-podinfo\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:56:15.363490 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.363289 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-workload-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:56:15.363490 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.363302 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zcwb\" (UniqueName: \"kubernetes.io/projected/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b-kube-api-access-7zcwb\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:56:15.371343 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.371306 2571 generic.go:356] "Generic (PLEG): container finished" podID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerID="e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3" exitCode=0 Jul 06 19:56:15.371343 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.371344 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" Jul 06 19:56:15.371574 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.371389 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" event={"ID":"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b","Type":"ContainerDied","Data":"e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3"} Jul 06 19:56:15.371574 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.371449 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k" event={"ID":"ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b","Type":"ContainerDied","Data":"a7cc321ef11d4392435e2633a5058ddfd8448b057f84a201efa1caa0cd6ab1f2"} Jul 06 19:56:15.371574 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.371478 2571 scope.go:117] "RemoveContainer" containerID="e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3" Jul 06 19:56:15.382987 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.382968 2571 scope.go:117] "RemoveContainer" containerID="04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15" Jul 06 19:56:15.391703 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.391684 2571 scope.go:117] "RemoveContainer" containerID="e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3" Jul 06 19:56:15.391983 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:56:15.391962 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3\": container with ID starting with e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3 not found: ID does not exist" containerID="e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3" Jul 06 19:56:15.392034 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.391993 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3"} err="failed to get container status \"e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3\": rpc error: code = NotFound desc = could not find container \"e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3\": container with ID starting with e86cb5f7ce7db955aebc67fb58ebf5e76800463fc6ed79becf284ba0e919c8a3 not found: ID does not exist" Jul 06 19:56:15.392034 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.392011 2571 scope.go:117] "RemoveContainer" containerID="04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15" Jul 06 19:56:15.392247 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:56:15.392228 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15\": container with ID starting with 04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15 not found: ID does not exist" containerID="04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15" Jul 06 19:56:15.392322 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.392256 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15"} err="failed to get container status \"04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15\": rpc error: code = NotFound desc = could not find container \"04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15\": container with ID starting with 04eaeee28f8b89fda392aa24e4c1337d4a9b31d5102c944d477fa98cd21fcd15 not found: ID does not exist" Jul 06 19:56:15.395993 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.395964 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k"] Jul 06 19:56:15.401300 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.401259 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bab069-openshift-default-85b68dbb5c-lh97k"] Jul 06 19:56:15.462209 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:15.462169 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" path="/var/lib/kubelet/pods/ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b/volumes" Jul 06 19:56:19.177579 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.177543 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn"] Jul 06 19:56:19.178189 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.178166 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" Jul 06 19:56:19.178189 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.178190 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" Jul 06 19:56:19.178390 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.178217 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" Jul 06 19:56:19.178390 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.178225 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" Jul 06 19:56:19.178390 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.178371 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" Jul 06 19:56:19.178390 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.178386 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee3dccaf-4e3f-4974-b21c-1bd5925f2c4b" containerName="istio-proxy" Jul 06 19:56:19.180543 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.180522 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.182963 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.182931 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:56:19.183104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.183019 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:56:19.183104 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.183048 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:56:19.183219 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.183054 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-97cecc-openshift-default-dockercfg-689ql\"" Jul 06 19:56:19.193494 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.193473 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn"] Jul 06 19:56:19.200218 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.200190 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-certs\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.200362 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.200225 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-credential-socket\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.200362 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.200255 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-envoy\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.200362 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.200322 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-data\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.200534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.200371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjms\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-kube-api-access-dsjms\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.200534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.200415 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a3f551-9926-45ef-bf97-090047b2885b-istiod-ca-cert\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.200534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.200436 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-socket\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.200534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.200480 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-istio-token\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.200534 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.200510 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a3f551-9926-45ef-bf97-090047b2885b-istio-podinfo\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301003 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.300964 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjms\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-kube-api-access-dsjms\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301170 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301020 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a3f551-9926-45ef-bf97-090047b2885b-istiod-ca-cert\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301170 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301135 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-socket\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301257 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301195 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-istio-token\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301257 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301238 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a3f551-9926-45ef-bf97-090047b2885b-istio-podinfo\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301307 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-certs\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-credential-socket\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301376 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301370 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-envoy\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301527 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301417 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-data\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301606 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301540 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-socket\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301715 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301676 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-certs\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301756 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-credential-socket\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301782 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a3f551-9926-45ef-bf97-090047b2885b-istiod-ca-cert\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.301848 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.301807 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-data\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.303693 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.303671 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-envoy\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.303941 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.303925 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a3f551-9926-45ef-bf97-090047b2885b-istio-podinfo\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.308621 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.308603 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-istio-token\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.308810 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.308795 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjms\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-kube-api-access-dsjms\") pod \"e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.491395 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.491303 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:19.633635 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.633562 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn"] Jul 06 19:56:19.635876 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:56:19.635843 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a3f551_9926_45ef_bf97_090047b2885b.slice/crio-adcf81b5368d4b2304ca9c1c8f10391b9753dba15b851a8b1960527faa0a5cf5 WatchSource:0}: Error finding container adcf81b5368d4b2304ca9c1c8f10391b9753dba15b851a8b1960527faa0a5cf5: Status 404 returned error can't find the container with id adcf81b5368d4b2304ca9c1c8f10391b9753dba15b851a8b1960527faa0a5cf5 Jul 06 19:56:19.638239 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.638204 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:56:19.638322 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.638289 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:56:19.638365 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:19.638324 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:56:20.398544 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:20.398502 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" event={"ID":"97a3f551-9926-45ef-bf97-090047b2885b","Type":"ContainerStarted","Data":"1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255"} Jul 06 19:56:20.399021 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:20.398562 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" event={"ID":"97a3f551-9926-45ef-bf97-090047b2885b","Type":"ContainerStarted","Data":"adcf81b5368d4b2304ca9c1c8f10391b9753dba15b851a8b1960527faa0a5cf5"} Jul 06 19:56:20.427671 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:20.427616 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" podStartSLOduration=1.427598507 podStartE2EDuration="1.427598507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:20.427573919 +0000 UTC m=+2167.487920845" watchObservedRunningTime="2026-07-06 19:56:20.427598507 +0000 UTC m=+2167.487945433" Jul 06 19:56:20.492366 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:20.492325 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:20.497169 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:20.497145 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:21.404157 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:21.404123 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:21.405121 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:21.405095 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:35.478065 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:35.477961 2571 generic.go:356] "Generic (PLEG): container finished" podID="97a3f551-9926-45ef-bf97-090047b2885b" containerID="1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255" exitCode=0 Jul 06 19:56:35.478065 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:35.478031 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" event={"ID":"97a3f551-9926-45ef-bf97-090047b2885b","Type":"ContainerDied","Data":"1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255"} Jul 06 19:56:35.478561 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:35.478494 2571 scope.go:117] "RemoveContainer" containerID="1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255" Jul 06 19:56:35.479549 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:35.479521 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:56:35.479640 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:35.479588 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:56:35.479640 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:35.479619 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:56:36.404062 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:36.404025 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:36.492051 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:36.492009 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" event={"ID":"97a3f551-9926-45ef-bf97-090047b2885b","Type":"ContainerStarted","Data":"8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0"} Jul 06 19:56:37.492349 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:37.492314 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:37.493263 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:37.493236 2571 patch_prober.go:28] interesting pod/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 06 19:56:37.493406 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:37.493311 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" podUID="97a3f551-9926-45ef-bf97-090047b2885b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 06 19:56:38.491906 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:38.491862 2571 patch_prober.go:28] interesting pod/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 06 19:56:38.492130 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:38.491924 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" podUID="97a3f551-9926-45ef-bf97-090047b2885b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 06 19:56:39.491861 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:39.491818 2571 patch_prober.go:28] interesting pod/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 06 19:56:39.492400 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:39.491889 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" podUID="97a3f551-9926-45ef-bf97-090047b2885b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 06 19:56:40.496052 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:40.496019 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:40.496461 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:40.496379 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:40.496993 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:40.496967 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 19:56:41.619192 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.619157 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b"] Jul 06 19:56:41.622358 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.622335 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.624717 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.624694 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9cb33a\"/\"openshift-service-ca.crt\"" Jul 06 19:56:41.625642 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.625618 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9cb33a\"/\"default-dockercfg-8ztn5\"" Jul 06 19:56:41.625763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.625655 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9cb33a\"/\"auth-test-model-9cb33a-kserve-self-signed-certs\"" Jul 06 19:56:41.625763 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.625666 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9cb33a\"/\"kube-root-ca.crt\"" Jul 06 19:56:41.636469 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.636444 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b"] Jul 06 19:56:41.729944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.729904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llf4p\" (UniqueName: \"kubernetes.io/projected/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-kube-api-access-llf4p\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.730145 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.729951 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tls-certs\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.730145 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.730029 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-model-cache\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.730145 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.730060 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tmp-dir\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.730145 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.730085 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-dshm\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.730339 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.730161 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-home\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.830971 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.830940 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llf4p\" (UniqueName: \"kubernetes.io/projected/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-kube-api-access-llf4p\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.830971 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.830977 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tls-certs\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.831214 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.831020 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-model-cache\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.831214 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.831049 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tmp-dir\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.831214 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.831071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-dshm\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.831214 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.831098 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-home\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.831512 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.831487 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-model-cache\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.831596 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.831527 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-home\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.831596 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.831547 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tmp-dir\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.833610 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.833590 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-dshm\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.833877 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.833855 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tls-certs\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.839892 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.839868 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llf4p\" (UniqueName: \"kubernetes.io/projected/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-kube-api-access-llf4p\") pod \"auth-test-model-9cb33a-kserve-6454476f9f-9rr7b\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:41.931790 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:41.931696 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:42.080074 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:42.080047 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b"] Jul 06 19:56:42.082463 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:56:42.082423 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77c25a88_d0ab_49f7_b1fc_dc87fdfbd8c8.slice/crio-029c24dee215dc2e56fe41f8d98bc90e3fca68c0f28213d6b47f65eb0ad33a6d WatchSource:0}: Error finding container 029c24dee215dc2e56fe41f8d98bc90e3fca68c0f28213d6b47f65eb0ad33a6d: Status 404 returned error can't find the container with id 029c24dee215dc2e56fe41f8d98bc90e3fca68c0f28213d6b47f65eb0ad33a6d Jul 06 19:56:42.523305 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:42.523258 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" event={"ID":"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8","Type":"ContainerStarted","Data":"adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f"} Jul 06 19:56:42.523494 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:42.523314 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" event={"ID":"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8","Type":"ContainerStarted","Data":"029c24dee215dc2e56fe41f8d98bc90e3fca68c0f28213d6b47f65eb0ad33a6d"} Jul 06 19:56:42.540033 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:42.539983 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" podStartSLOduration=1.539968001 podStartE2EDuration="1.539968001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:42.538917071 +0000 UTC m=+2189.599263997" watchObservedRunningTime="2026-07-06 19:56:42.539968001 +0000 UTC m=+2189.600314927" Jul 06 19:56:51.932050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:51.932004 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:51.932050 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:51.932060 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:51.944634 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:51.944606 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:56:52.584452 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:56:52.584421 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:58:57.008393 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:58:57.008358 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b"] Jul 06 19:59:09.225661 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.225629 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt"] Jul 06 19:59:09.229666 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.229644 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.231997 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.231978 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9cb33a\"/\"sub-test-model-9cb33a-kserve-self-signed-certs\"" Jul 06 19:59:09.238762 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.238738 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt"] Jul 06 19:59:09.339336 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.339305 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-model-cache\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.339336 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.339343 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a00ba95-ebb8-4d68-8e76-fd3856584420-tls-certs\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.339543 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.339450 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-tmp-dir\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.339543 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.339486 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cjx9\" (UniqueName: \"kubernetes.io/projected/2a00ba95-ebb8-4d68-8e76-fd3856584420-kube-api-access-8cjx9\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.339636 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.339589 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-home\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.339636 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.339619 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-dshm\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.440885 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.440850 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-model-cache\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.440885 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.440890 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a00ba95-ebb8-4d68-8e76-fd3856584420-tls-certs\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.441126 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.440979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-tmp-dir\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.441126 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.441011 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cjx9\" (UniqueName: \"kubernetes.io/projected/2a00ba95-ebb8-4d68-8e76-fd3856584420-kube-api-access-8cjx9\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.441126 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.441060 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-home\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.441126 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.441105 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-dshm\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.441401 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.441377 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-model-cache\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.441464 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.441430 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-tmp-dir\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.441507 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.441478 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-home\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.443443 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.443423 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-dshm\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.443621 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.443605 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a00ba95-ebb8-4d68-8e76-fd3856584420-tls-certs\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.450643 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.450622 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cjx9\" (UniqueName: \"kubernetes.io/projected/2a00ba95-ebb8-4d68-8e76-fd3856584420-kube-api-access-8cjx9\") pod \"sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.539390 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.539347 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:09.692483 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:09.692451 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt"] Jul 06 19:59:09.696220 ip-10-0-131-211 kubenswrapper[2571]: W0706 19:59:09.696194 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a00ba95_ebb8_4d68_8e76_fd3856584420.slice/crio-db4517f56cba1de48f20ca5e74c2367281b063fe2c85bd56060bd9f8a652b068 WatchSource:0}: Error finding container db4517f56cba1de48f20ca5e74c2367281b063fe2c85bd56060bd9f8a652b068: Status 404 returned error can't find the container with id db4517f56cba1de48f20ca5e74c2367281b063fe2c85bd56060bd9f8a652b068 Jul 06 19:59:10.291801 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:10.291764 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" event={"ID":"2a00ba95-ebb8-4d68-8e76-fd3856584420","Type":"ContainerStarted","Data":"993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60"} Jul 06 19:59:10.292167 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:10.291810 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" event={"ID":"2a00ba95-ebb8-4d68-8e76-fd3856584420","Type":"ContainerStarted","Data":"db4517f56cba1de48f20ca5e74c2367281b063fe2c85bd56060bd9f8a652b068"} Jul 06 19:59:10.315028 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:10.314969 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" podStartSLOduration=1.314953276 podStartE2EDuration="1.314953276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:10.314432382 +0000 UTC m=+2337.374779308" watchObservedRunningTime="2026-07-06 19:59:10.314953276 +0000 UTC m=+2337.375300202" Jul 06 19:59:12.031824 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.031768 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" podUID="77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" containerName="main" containerID="cri-o://adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f" gracePeriod=45 Jul 06 19:59:12.295092 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.295031 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:59:12.303170 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.303144 2571 generic.go:356] "Generic (PLEG): container finished" podID="77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" containerID="adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f" exitCode=0 Jul 06 19:59:12.303259 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.303175 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" Jul 06 19:59:12.303259 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.303225 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" event={"ID":"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8","Type":"ContainerDied","Data":"adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f"} Jul 06 19:59:12.303370 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.303293 2571 scope.go:117] "RemoveContainer" containerID="adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f" Jul 06 19:59:12.303370 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.303293 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b" event={"ID":"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8","Type":"ContainerDied","Data":"029c24dee215dc2e56fe41f8d98bc90e3fca68c0f28213d6b47f65eb0ad33a6d"} Jul 06 19:59:12.312722 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.312696 2571 scope.go:117] "RemoveContainer" containerID="adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f" Jul 06 19:59:12.313037 ip-10-0-131-211 kubenswrapper[2571]: E0706 19:59:12.313012 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f\": container with ID starting with adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f not found: ID does not exist" containerID="adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f" Jul 06 19:59:12.313187 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.313067 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f"} err="failed to get container status \"adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f\": rpc error: code = NotFound desc = could not find container \"adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f\": container with ID starting with adfe0fc54ebeca24633b28bc6a84dd43340c77ef47c45a65c66f043572b2f88f not found: ID does not exist" Jul 06 19:59:12.367244 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367208 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-model-cache\") pod \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " Jul 06 19:59:12.367388 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367338 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tmp-dir\") pod \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " Jul 06 19:59:12.367388 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367376 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-home\") pod \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " Jul 06 19:59:12.367476 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367426 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tls-certs\") pod \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " Jul 06 19:59:12.367536 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367496 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llf4p\" (UniqueName: \"kubernetes.io/projected/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-kube-api-access-llf4p\") pod \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " Jul 06 19:59:12.367536 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367526 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-dshm\") pod \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\" (UID: \"77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8\") " Jul 06 19:59:12.367818 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367521 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" (UID: "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:12.367818 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367610 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" (UID: "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:12.367818 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367686 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-home" (OuterVolumeSpecName: "home") pod "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" (UID: "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:12.368037 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367876 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-home\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.368037 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367893 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-model-cache\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.368037 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.367906 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tmp-dir\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.369806 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.369777 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" (UID: "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:59:12.369918 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.369809 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-dshm" (OuterVolumeSpecName: "dshm") pod "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" (UID: "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:12.370031 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.370009 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-kube-api-access-llf4p" (OuterVolumeSpecName: "kube-api-access-llf4p") pod "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" (UID: "77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8"). InnerVolumeSpecName "kube-api-access-llf4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:12.468758 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.468723 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-tls-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.468758 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.468755 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llf4p\" (UniqueName: \"kubernetes.io/projected/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-kube-api-access-llf4p\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.468944 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.468767 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8-dshm\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.624310 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.624258 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b"] Jul 06 19:59:12.628237 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:12.628214 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/auth-test-model-9cb33a-kserve-6454476f9f-9rr7b"] Jul 06 19:59:13.462690 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:13.462656 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" path="/var/lib/kubelet/pods/77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8/volumes" Jul 06 19:59:19.540023 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:19.539993 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:19.540515 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:19.540034 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:19.552067 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:19.552046 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 19:59:20.361443 ip-10-0-131-211 kubenswrapper[2571]: I0706 19:59:20.361410 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 20:00:28.116895 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:28.116862 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt"] Jul 06 20:00:43.137028 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.136912 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" podUID="2a00ba95-ebb8-4d68-8e76-fd3856584420" containerName="main" containerID="cri-o://993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60" gracePeriod=45 Jul 06 20:00:43.400946 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.400883 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 20:00:43.525255 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525230 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a00ba95-ebb8-4d68-8e76-fd3856584420-tls-certs\") pod \"2a00ba95-ebb8-4d68-8e76-fd3856584420\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " Jul 06 20:00:43.525435 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525301 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-model-cache\") pod \"2a00ba95-ebb8-4d68-8e76-fd3856584420\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " Jul 06 20:00:43.525435 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525341 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-dshm\") pod \"2a00ba95-ebb8-4d68-8e76-fd3856584420\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " Jul 06 20:00:43.525435 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525389 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-home\") pod \"2a00ba95-ebb8-4d68-8e76-fd3856584420\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " Jul 06 20:00:43.525435 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525430 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cjx9\" (UniqueName: \"kubernetes.io/projected/2a00ba95-ebb8-4d68-8e76-fd3856584420-kube-api-access-8cjx9\") pod \"2a00ba95-ebb8-4d68-8e76-fd3856584420\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " Jul 06 20:00:43.525645 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525482 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-tmp-dir\") pod \"2a00ba95-ebb8-4d68-8e76-fd3856584420\" (UID: \"2a00ba95-ebb8-4d68-8e76-fd3856584420\") " Jul 06 20:00:43.525645 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525595 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a00ba95-ebb8-4d68-8e76-fd3856584420" (UID: "2a00ba95-ebb8-4d68-8e76-fd3856584420"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:43.525756 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525688 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-home" (OuterVolumeSpecName: "home") pod "2a00ba95-ebb8-4d68-8e76-fd3856584420" (UID: "2a00ba95-ebb8-4d68-8e76-fd3856584420"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:43.525756 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525750 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-model-cache\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:00:43.525857 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.525805 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a00ba95-ebb8-4d68-8e76-fd3856584420" (UID: "2a00ba95-ebb8-4d68-8e76-fd3856584420"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:43.527506 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.527479 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-dshm" (OuterVolumeSpecName: "dshm") pod "2a00ba95-ebb8-4d68-8e76-fd3856584420" (UID: "2a00ba95-ebb8-4d68-8e76-fd3856584420"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:43.527605 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.527540 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a00ba95-ebb8-4d68-8e76-fd3856584420-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a00ba95-ebb8-4d68-8e76-fd3856584420" (UID: "2a00ba95-ebb8-4d68-8e76-fd3856584420"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:00:43.527668 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.527649 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a00ba95-ebb8-4d68-8e76-fd3856584420-kube-api-access-8cjx9" (OuterVolumeSpecName: "kube-api-access-8cjx9") pod "2a00ba95-ebb8-4d68-8e76-fd3856584420" (UID: "2a00ba95-ebb8-4d68-8e76-fd3856584420"). InnerVolumeSpecName "kube-api-access-8cjx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:43.626410 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.626385 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a00ba95-ebb8-4d68-8e76-fd3856584420-tls-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:00:43.626410 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.626407 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-dshm\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:00:43.626410 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.626416 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-home\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:00:43.626616 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.626425 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cjx9\" (UniqueName: \"kubernetes.io/projected/2a00ba95-ebb8-4d68-8e76-fd3856584420-kube-api-access-8cjx9\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:00:43.626616 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.626434 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a00ba95-ebb8-4d68-8e76-fd3856584420-tmp-dir\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:00:43.779224 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.779191 2571 generic.go:356] "Generic (PLEG): container finished" podID="2a00ba95-ebb8-4d68-8e76-fd3856584420" containerID="993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60" exitCode=0 Jul 06 20:00:43.779390 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.779233 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" Jul 06 20:00:43.779390 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.779296 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" event={"ID":"2a00ba95-ebb8-4d68-8e76-fd3856584420","Type":"ContainerDied","Data":"993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60"} Jul 06 20:00:43.779390 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.779343 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt" event={"ID":"2a00ba95-ebb8-4d68-8e76-fd3856584420","Type":"ContainerDied","Data":"db4517f56cba1de48f20ca5e74c2367281b063fe2c85bd56060bd9f8a652b068"} Jul 06 20:00:43.779390 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.779360 2571 scope.go:117] "RemoveContainer" containerID="993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60" Jul 06 20:00:43.789355 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.789338 2571 scope.go:117] "RemoveContainer" containerID="993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60" Jul 06 20:00:43.789613 ip-10-0-131-211 kubenswrapper[2571]: E0706 20:00:43.789591 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60\": container with ID starting with 993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60 not found: ID does not exist" containerID="993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60" Jul 06 20:00:43.789729 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.789625 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60"} err="failed to get container status \"993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60\": rpc error: code = NotFound desc = could not find container \"993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60\": container with ID starting with 993bbffa2a624cc55c54bc1d41e8d1a09eecd3d3c07067580680c89b9adc4c60 not found: ID does not exist" Jul 06 20:00:43.803755 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.803733 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt"] Jul 06 20:00:43.810005 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:43.809984 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/sub-test-model-9cb33a-kserve-d99dfc6d5-tvkxt"] Jul 06 20:00:45.470011 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:45.469808 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a00ba95-ebb8-4d68-8e76-fd3856584420" path="/var/lib/kubelet/pods/2a00ba95-ebb8-4d68-8e76-fd3856584420/volumes" Jul 06 20:00:55.231975 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.231938 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd"] Jul 06 20:00:55.232424 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.232360 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a00ba95-ebb8-4d68-8e76-fd3856584420" containerName="main" Jul 06 20:00:55.232424 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.232371 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a00ba95-ebb8-4d68-8e76-fd3856584420" containerName="main" Jul 06 20:00:55.232424 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.232398 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" containerName="main" Jul 06 20:00:55.232424 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.232403 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" containerName="main" Jul 06 20:00:55.232566 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.232462 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c25a88-d0ab-49f7-b1fc-dc87fdfbd8c8" containerName="main" Jul 06 20:00:55.232566 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.232469 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a00ba95-ebb8-4d68-8e76-fd3856584420" containerName="main" Jul 06 20:00:55.235537 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.235518 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.239229 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.239210 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9cb33a\"/\"kube-root-ca.crt\"" Jul 06 20:00:55.239374 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.239247 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9cb33a\"/\"openshift-service-ca.crt\"" Jul 06 20:00:55.239374 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.239349 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9cb33a\"/\"default-dockercfg-8ztn5\"" Jul 06 20:00:55.239374 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.239349 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9cb33a\"/\"rate-test-model-9cb33a-kserve-self-signed-certs\"" Jul 06 20:00:55.246720 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.246699 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd"] Jul 06 20:00:55.339507 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.339476 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-dshm\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.339667 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.339525 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tmp-dir\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.339667 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.339550 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tls-certs\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.339667 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.339589 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-home\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.339667 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.339642 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-model-cache\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.339870 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.339720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4hr\" (UniqueName: \"kubernetes.io/projected/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-kube-api-access-gw4hr\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.440283 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.440246 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-model-cache\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.440435 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.440326 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4hr\" (UniqueName: \"kubernetes.io/projected/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-kube-api-access-gw4hr\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.440435 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.440386 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-dshm\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.440435 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.440419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tmp-dir\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.440580 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.440439 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tls-certs\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.440580 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.440463 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-home\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.440696 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.440673 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-model-cache\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.440774 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.440753 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tmp-dir\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.440836 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.440782 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-home\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.442847 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.442826 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-dshm\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.443004 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.442986 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tls-certs\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.449734 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.449711 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4hr\" (UniqueName: \"kubernetes.io/projected/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-kube-api-access-gw4hr\") pod \"rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.545144 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.545110 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:00:55.691850 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.691822 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd"] Jul 06 20:00:55.693999 ip-10-0-131-211 kubenswrapper[2571]: W0706 20:00:55.693973 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b0db6a9_77e5_49ad_b1ba_64ecb91bb0f2.slice/crio-0a3a92ec6ce2072c0ce88a50b56929c20bceaa4bd045ab5bd6c1b7b6c3d1aa12 WatchSource:0}: Error finding container 0a3a92ec6ce2072c0ce88a50b56929c20bceaa4bd045ab5bd6c1b7b6c3d1aa12: Status 404 returned error can't find the container with id 0a3a92ec6ce2072c0ce88a50b56929c20bceaa4bd045ab5bd6c1b7b6c3d1aa12 Jul 06 20:00:55.841189 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.841086 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" event={"ID":"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2","Type":"ContainerStarted","Data":"77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6"} Jul 06 20:00:55.841189 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:00:55.841138 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" event={"ID":"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2","Type":"ContainerStarted","Data":"0a3a92ec6ce2072c0ce88a50b56929c20bceaa4bd045ab5bd6c1b7b6c3d1aa12"} Jul 06 20:01:05.546332 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:01:05.546297 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:01:05.546836 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:01:05.546349 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:01:05.559557 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:01:05.559529 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:01:05.586618 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:01:05.586564 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" podStartSLOduration=10.58654101 podStartE2EDuration="10.58654101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:55.864044525 +0000 UTC m=+2442.924391451" watchObservedRunningTime="2026-07-06 20:01:05.58654101 +0000 UTC m=+2452.646887937" Jul 06 20:01:05.904108 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:01:05.904025 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:02:05.338483 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:05.338402 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd"] Jul 06 20:02:20.359969 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.359907 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" podUID="3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" containerName="main" containerID="cri-o://77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6" gracePeriod=45 Jul 06 20:02:20.617538 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.617486 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:02:20.722078 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722045 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-dshm\") pod \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " Jul 06 20:02:20.722218 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722137 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw4hr\" (UniqueName: \"kubernetes.io/projected/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-kube-api-access-gw4hr\") pod \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " Jul 06 20:02:20.722218 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722159 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tmp-dir\") pod \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " Jul 06 20:02:20.722218 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722197 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tls-certs\") pod \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " Jul 06 20:02:20.722437 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722242 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-home\") pod \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " Jul 06 20:02:20.722437 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722312 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-model-cache\") pod \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\" (UID: \"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2\") " Jul 06 20:02:20.722561 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722538 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" (UID: "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:20.722635 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722554 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-home" (OuterVolumeSpecName: "home") pod "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" (UID: "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:20.722699 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722635 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" (UID: "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:20.722699 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.722643 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tmp-dir\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:20.724368 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.724343 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-dshm" (OuterVolumeSpecName: "dshm") pod "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" (UID: "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:20.724467 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.724387 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" (UID: "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:02:20.724467 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.724403 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-kube-api-access-gw4hr" (OuterVolumeSpecName: "kube-api-access-gw4hr") pod "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" (UID: "3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2"). InnerVolumeSpecName "kube-api-access-gw4hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:20.823600 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.823572 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-tls-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:20.823600 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.823596 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-home\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:20.823600 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.823605 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-model-cache\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:20.823808 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.823613 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-dshm\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:20.823808 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:20.823620 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw4hr\" (UniqueName: \"kubernetes.io/projected/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2-kube-api-access-gw4hr\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:21.279846 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.279810 2571 generic.go:356] "Generic (PLEG): container finished" podID="3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" containerID="77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6" exitCode=0 Jul 06 20:02:21.280068 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.279871 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" Jul 06 20:02:21.280068 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.279874 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" event={"ID":"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2","Type":"ContainerDied","Data":"77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6"} Jul 06 20:02:21.280068 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.280003 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd" event={"ID":"3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2","Type":"ContainerDied","Data":"0a3a92ec6ce2072c0ce88a50b56929c20bceaa4bd045ab5bd6c1b7b6c3d1aa12"} Jul 06 20:02:21.280068 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.280031 2571 scope.go:117] "RemoveContainer" containerID="77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6" Jul 06 20:02:21.289622 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.289606 2571 scope.go:117] "RemoveContainer" containerID="77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6" Jul 06 20:02:21.289847 ip-10-0-131-211 kubenswrapper[2571]: E0706 20:02:21.289831 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6\": container with ID starting with 77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6 not found: ID does not exist" containerID="77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6" Jul 06 20:02:21.289896 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.289854 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6"} err="failed to get container status \"77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6\": rpc error: code = NotFound desc = could not find container \"77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6\": container with ID starting with 77e5b9d7648bbfb1f382690c165b92348c9bc561cb3088c8e12ab9ce197f8da6 not found: ID does not exist" Jul 06 20:02:21.301334 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.301301 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd"] Jul 06 20:02:21.304263 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.304241 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9cb33a/rate-test-model-9cb33a-kserve-dd99cc6ff-m2rrd"] Jul 06 20:02:21.462264 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:21.462228 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" path="/var/lib/kubelet/pods/3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2/volumes" Jul 06 20:02:31.295563 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:31.295530 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn"] Jul 06 20:02:31.296007 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:31.295747 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" podUID="97a3f551-9926-45ef-bf97-090047b2885b" containerName="istio-proxy" containerID="cri-o://8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0" gracePeriod=30 Jul 06 20:02:36.423157 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.423125 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2xttg_4694615b-ae46-44cd-8a55-2f1b46d595ee/manager/0.log" Jul 06 20:02:36.540967 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.540943 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bq6ll_7d2ac511-e78f-4dfc-adb9-205defb7f9f5/manager/0.log" Jul 06 20:02:36.561138 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.561089 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 20:02:36.674663 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.674632 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-certs\") pod \"97a3f551-9926-45ef-bf97-090047b2885b\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " Jul 06 20:02:36.674858 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.674687 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-data\") pod \"97a3f551-9926-45ef-bf97-090047b2885b\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " Jul 06 20:02:36.674858 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.674725 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsjms\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-kube-api-access-dsjms\") pod \"97a3f551-9926-45ef-bf97-090047b2885b\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " Jul 06 20:02:36.674858 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.674768 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a3f551-9926-45ef-bf97-090047b2885b-istio-podinfo\") pod \"97a3f551-9926-45ef-bf97-090047b2885b\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " Jul 06 20:02:36.674858 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.674805 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a3f551-9926-45ef-bf97-090047b2885b-istiod-ca-cert\") pod \"97a3f551-9926-45ef-bf97-090047b2885b\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " Jul 06 20:02:36.675048 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.674934 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-istio-token\") pod \"97a3f551-9926-45ef-bf97-090047b2885b\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " Jul 06 20:02:36.675048 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.674956 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97a3f551-9926-45ef-bf97-090047b2885b" (UID: "97a3f551-9926-45ef-bf97-090047b2885b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:36.675048 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.674970 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-credential-socket\") pod \"97a3f551-9926-45ef-bf97-090047b2885b\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " Jul 06 20:02:36.675048 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.675019 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-socket\") pod \"97a3f551-9926-45ef-bf97-090047b2885b\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " Jul 06 20:02:36.675048 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.675047 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-envoy\") pod \"97a3f551-9926-45ef-bf97-090047b2885b\" (UID: \"97a3f551-9926-45ef-bf97-090047b2885b\") " Jul 06 20:02:36.675336 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.675241 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a3f551-9926-45ef-bf97-090047b2885b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97a3f551-9926-45ef-bf97-090047b2885b" (UID: "97a3f551-9926-45ef-bf97-090047b2885b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:02:36.675481 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.675456 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97a3f551-9926-45ef-bf97-090047b2885b" (UID: "97a3f551-9926-45ef-bf97-090047b2885b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:36.675585 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.675572 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a3f551-9926-45ef-bf97-090047b2885b-istiod-ca-cert\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:36.675652 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.675591 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-credential-socket\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:36.675652 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.675606 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-certs\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:36.675797 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.675771 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97a3f551-9926-45ef-bf97-090047b2885b" (UID: "97a3f551-9926-45ef-bf97-090047b2885b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:36.675876 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.675733 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-data" (OuterVolumeSpecName: "istio-data") pod "97a3f551-9926-45ef-bf97-090047b2885b" (UID: "97a3f551-9926-45ef-bf97-090047b2885b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:36.677371 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.677345 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97a3f551-9926-45ef-bf97-090047b2885b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97a3f551-9926-45ef-bf97-090047b2885b" (UID: "97a3f551-9926-45ef-bf97-090047b2885b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:02:36.677486 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.677441 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-istio-token" (OuterVolumeSpecName: "istio-token") pod "97a3f551-9926-45ef-bf97-090047b2885b" (UID: "97a3f551-9926-45ef-bf97-090047b2885b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:36.677691 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.677669 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-kube-api-access-dsjms" (OuterVolumeSpecName: "kube-api-access-dsjms") pod "97a3f551-9926-45ef-bf97-090047b2885b" (UID: "97a3f551-9926-45ef-bf97-090047b2885b"). InnerVolumeSpecName "kube-api-access-dsjms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:36.677979 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.677958 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97a3f551-9926-45ef-bf97-090047b2885b" (UID: "97a3f551-9926-45ef-bf97-090047b2885b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:36.776616 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.776585 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-workload-socket\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:36.776616 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.776608 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-envoy\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:36.776616 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.776617 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a3f551-9926-45ef-bf97-090047b2885b-istio-data\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:36.776822 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.776625 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsjms\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-kube-api-access-dsjms\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:36.776822 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.776633 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a3f551-9926-45ef-bf97-090047b2885b-istio-podinfo\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:36.776822 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:36.776641 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a3f551-9926-45ef-bf97-090047b2885b-istio-token\") on node \"ip-10-0-131-211.ec2.internal\" DevicePath \"\"" Jul 06 20:02:37.363453 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.363411 2571 generic.go:356] "Generic (PLEG): container finished" podID="97a3f551-9926-45ef-bf97-090047b2885b" containerID="8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0" exitCode=0 Jul 06 20:02:37.363637 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.363456 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" Jul 06 20:02:37.363637 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.363487 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" event={"ID":"97a3f551-9926-45ef-bf97-090047b2885b","Type":"ContainerDied","Data":"8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0"} Jul 06 20:02:37.363637 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.363543 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn" event={"ID":"97a3f551-9926-45ef-bf97-090047b2885b","Type":"ContainerDied","Data":"adcf81b5368d4b2304ca9c1c8f10391b9753dba15b851a8b1960527faa0a5cf5"} Jul 06 20:02:37.363637 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.363574 2571 scope.go:117] "RemoveContainer" containerID="8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0" Jul 06 20:02:37.373481 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.373462 2571 scope.go:117] "RemoveContainer" containerID="1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255" Jul 06 20:02:37.382107 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.382085 2571 scope.go:117] "RemoveContainer" containerID="8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0" Jul 06 20:02:37.382416 ip-10-0-131-211 kubenswrapper[2571]: E0706 20:02:37.382396 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0\": container with ID starting with 8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0 not found: ID does not exist" containerID="8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0" Jul 06 20:02:37.382480 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.382425 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0"} err="failed to get container status \"8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0\": rpc error: code = NotFound desc = could not find container \"8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0\": container with ID starting with 8cc7ce6f6fd11840202923f48c5e3b0bc8850030376f211916a514412f17d1b0 not found: ID does not exist" Jul 06 20:02:37.382480 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.382442 2571 scope.go:117] "RemoveContainer" containerID="1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255" Jul 06 20:02:37.382739 ip-10-0-131-211 kubenswrapper[2571]: E0706 20:02:37.382718 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255\": container with ID starting with 1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255 not found: ID does not exist" containerID="1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255" Jul 06 20:02:37.382791 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.382745 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255"} err="failed to get container status \"1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255\": rpc error: code = NotFound desc = could not find container \"1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255\": container with ID starting with 1461dc6a2c433e17b13390548686b29cd1501244113b446cbcd8904f372f8255 not found: ID does not exist" Jul 06 20:02:37.388606 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.388581 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn"] Jul 06 20:02:37.394508 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.394487 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-97cecc-openshift-default-688cdd68cf-szfjn"] Jul 06 20:02:37.456648 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.456626 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69fd78d8d5-7bxvm_cdf137ce-5324-4116-906d-a8ee2b92c24b/manager/0.log" Jul 06 20:02:37.462569 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.462545 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a3f551-9926-45ef-bf97-090047b2885b" path="/var/lib/kubelet/pods/97a3f551-9926-45ef-bf97-090047b2885b/volumes" Jul 06 20:02:37.470717 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:37.470687 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69fd78d8d5-7bxvm_cdf137ce-5324-4116-906d-a8ee2b92c24b/copy-manifests/0.log" Jul 06 20:02:39.198655 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:39.198621 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-959dj_c02f3250-b02f-4b77-aa10-2d9d80a1298c/manager/0.log" Jul 06 20:02:39.530502 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:39.530476 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cxrmx_9947ecb8-5a9d-401a-80be-8cc3fb3e7e73/registry-server/0.log" Jul 06 20:02:39.651780 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:39.651752 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hpzdh_45f796af-845b-47a2-9b37-e3186c4ddf66/manager/0.log" Jul 06 20:02:39.879411 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:39.879344 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ggtmv_7d267ed6-3497-411c-a842-2f42cc120f8d/manager/0.log" Jul 06 20:02:41.589833 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:41.589809 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6dkls_4fcf5aba-77ef-42a0-bfe4-0f9fbd759dfe/main/0.log" Jul 06 20:02:41.701172 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:41.701149 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr8zpc_13d46d60-dc29-41fc-9004-e931dccd63c3/main/0.log" Jul 06 20:02:49.393889 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:49.393858 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8rchd_44d4a5a8-5d12-483d-939c-5e40d5d3658b/global-pull-secret-syncer/0.log" Jul 06 20:02:49.584167 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:49.584137 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nfhbg_5d99aa6b-369e-4f8e-984f-1b3097abfbcf/konnectivity-agent/0.log" Jul 06 20:02:49.669937 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:49.669865 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-211.ec2.internal_cc1d303b9ebb2e2da2b7750642266d77/haproxy/0.log" Jul 06 20:02:54.904324 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:54.904286 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-959dj_c02f3250-b02f-4b77-aa10-2d9d80a1298c/manager/0.log" Jul 06 20:02:55.023009 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:55.022978 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cxrmx_9947ecb8-5a9d-401a-80be-8cc3fb3e7e73/registry-server/0.log" Jul 06 20:02:55.294838 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:55.294800 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hpzdh_45f796af-845b-47a2-9b37-e3186c4ddf66/manager/0.log" Jul 06 20:02:55.398427 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:55.398353 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ggtmv_7d267ed6-3497-411c-a842-2f42cc120f8d/manager/0.log" Jul 06 20:02:57.183374 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.183349 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06a4f69b-68d3-475f-ab99-f38386c1383b/alertmanager/0.log" Jul 06 20:02:57.205736 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.205713 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06a4f69b-68d3-475f-ab99-f38386c1383b/config-reloader/0.log" Jul 06 20:02:57.229564 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.229537 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06a4f69b-68d3-475f-ab99-f38386c1383b/kube-rbac-proxy-web/0.log" Jul 06 20:02:57.253823 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.253801 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06a4f69b-68d3-475f-ab99-f38386c1383b/kube-rbac-proxy/0.log" Jul 06 20:02:57.276329 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.276306 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06a4f69b-68d3-475f-ab99-f38386c1383b/kube-rbac-proxy-metric/0.log" Jul 06 20:02:57.299696 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.299677 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06a4f69b-68d3-475f-ab99-f38386c1383b/prom-label-proxy/0.log" Jul 06 20:02:57.323428 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.323406 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06a4f69b-68d3-475f-ab99-f38386c1383b/init-config-reloader/0.log" Jul 06 20:02:57.401581 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.401518 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-vbg87_ce527849-225c-4e19-ae07-7e40bc516e3c/kube-state-metrics/0.log" Jul 06 20:02:57.423792 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.423772 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-vbg87_ce527849-225c-4e19-ae07-7e40bc516e3c/kube-rbac-proxy-main/0.log" Jul 06 20:02:57.449539 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.449511 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-vbg87_ce527849-225c-4e19-ae07-7e40bc516e3c/kube-rbac-proxy-self/0.log" Jul 06 20:02:57.481672 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.481650 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bf5f9c4b9-kvwtp_a8ef47f4-f4f0-48d1-a09b-89f6e6bbcacf/metrics-server/0.log" Jul 06 20:02:57.511254 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.511223 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-cswwx_08c932a1-c259-48fd-9f3b-65c1c51a536f/monitoring-plugin/0.log" Jul 06 20:02:57.642293 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.642253 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-94jtt_8434d9ae-3571-4d23-873c-42ea6176d06e/node-exporter/0.log" Jul 06 20:02:57.666072 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.665998 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-94jtt_8434d9ae-3571-4d23-873c-42ea6176d06e/kube-rbac-proxy/0.log" Jul 06 20:02:57.692033 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.692014 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-94jtt_8434d9ae-3571-4d23-873c-42ea6176d06e/init-textfile/0.log" Jul 06 20:02:57.826404 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.826379 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-8tqj6_5eefc10c-741b-4dad-b05b-0ac34b17d3c1/kube-rbac-proxy-main/0.log" Jul 06 20:02:57.852141 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.852121 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-8tqj6_5eefc10c-741b-4dad-b05b-0ac34b17d3c1/kube-rbac-proxy-self/0.log" Jul 06 20:02:57.878007 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.877986 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-8tqj6_5eefc10c-741b-4dad-b05b-0ac34b17d3c1/openshift-state-metrics/0.log" Jul 06 20:02:57.924308 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.924260 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f310f78-62c3-47fa-81a1-bfb782e93d1c/prometheus/0.log" Jul 06 20:02:57.959739 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.959721 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f310f78-62c3-47fa-81a1-bfb782e93d1c/config-reloader/0.log" Jul 06 20:02:57.989107 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:57.989088 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f310f78-62c3-47fa-81a1-bfb782e93d1c/thanos-sidecar/0.log" Jul 06 20:02:58.015547 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.015531 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f310f78-62c3-47fa-81a1-bfb782e93d1c/kube-rbac-proxy-web/0.log" Jul 06 20:02:58.039997 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.039978 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f310f78-62c3-47fa-81a1-bfb782e93d1c/kube-rbac-proxy/0.log" Jul 06 20:02:58.063573 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.063555 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f310f78-62c3-47fa-81a1-bfb782e93d1c/kube-rbac-proxy-thanos/0.log" Jul 06 20:02:58.092355 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.092328 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8f310f78-62c3-47fa-81a1-bfb782e93d1c/init-config-reloader/0.log" Jul 06 20:02:58.126160 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.126137 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-25qwk_4b87c79d-e82e-419a-8145-3d6c0e562304/prometheus-operator/0.log" Jul 06 20:02:58.146036 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.146017 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-25qwk_4b87c79d-e82e-419a-8145-3d6c0e562304/kube-rbac-proxy/0.log" Jul 06 20:02:58.205539 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.205487 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64bd95bbdb-gs77h_088fd4a9-6953-4809-bc48-d98f20202fe8/telemeter-client/0.log" Jul 06 20:02:58.237250 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.237231 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64bd95bbdb-gs77h_088fd4a9-6953-4809-bc48-d98f20202fe8/reload/0.log" Jul 06 20:02:58.278740 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.278724 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64bd95bbdb-gs77h_088fd4a9-6953-4809-bc48-d98f20202fe8/kube-rbac-proxy/0.log" Jul 06 20:02:58.322330 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.322314 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54f87764f-gc958_a1e27927-a3fb-4e73-854c-0175744772a6/thanos-query/0.log" Jul 06 20:02:58.347616 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.347602 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54f87764f-gc958_a1e27927-a3fb-4e73-854c-0175744772a6/kube-rbac-proxy-web/0.log" Jul 06 20:02:58.373443 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.373424 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54f87764f-gc958_a1e27927-a3fb-4e73-854c-0175744772a6/kube-rbac-proxy/0.log" Jul 06 20:02:58.399430 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.399408 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54f87764f-gc958_a1e27927-a3fb-4e73-854c-0175744772a6/prom-label-proxy/0.log" Jul 06 20:02:58.445821 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.445798 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54f87764f-gc958_a1e27927-a3fb-4e73-854c-0175744772a6/kube-rbac-proxy-rules/0.log" Jul 06 20:02:58.489464 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:58.489396 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54f87764f-gc958_a1e27927-a3fb-4e73-854c-0175744772a6/kube-rbac-proxy-metrics/0.log" Jul 06 20:02:59.724389 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.724355 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp"] Jul 06 20:02:59.725007 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.724986 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a3f551-9926-45ef-bf97-090047b2885b" containerName="istio-proxy" Jul 06 20:02:59.725120 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.725008 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a3f551-9926-45ef-bf97-090047b2885b" containerName="istio-proxy" Jul 06 20:02:59.725120 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.725021 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" containerName="main" Jul 06 20:02:59.725120 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.725029 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" containerName="main" Jul 06 20:02:59.725302 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.725184 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a3f551-9926-45ef-bf97-090047b2885b" containerName="istio-proxy" Jul 06 20:02:59.725302 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.725198 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a3f551-9926-45ef-bf97-090047b2885b" containerName="istio-proxy" Jul 06 20:02:59.725302 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.725209 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0db6a9-77e5-49ad-b1ba-64ecb91bb0f2" containerName="main" Jul 06 20:02:59.728631 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.728611 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.732366 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.732349 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sxl92\"/\"openshift-service-ca.crt\"" Jul 06 20:02:59.733522 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.733499 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sxl92\"/\"default-dockercfg-jrspn\"" Jul 06 20:02:59.733595 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.733499 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sxl92\"/\"kube-root-ca.crt\"" Jul 06 20:02:59.743735 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.743714 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp"] Jul 06 20:02:59.881890 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.881861 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-lib-modules\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.882069 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.881898 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hc8\" (UniqueName: \"kubernetes.io/projected/c8a00e7a-440d-48df-af38-51b80509ef4e-kube-api-access-s7hc8\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.882069 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.881916 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-sys\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.882069 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.881988 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-proc\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.882230 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.882089 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-podres\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.982701 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.982628 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hc8\" (UniqueName: \"kubernetes.io/projected/c8a00e7a-440d-48df-af38-51b80509ef4e-kube-api-access-s7hc8\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.982701 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.982657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-sys\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.982701 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.982685 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-proc\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.982952 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.982761 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-proc\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.982952 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.982763 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-sys\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.982952 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.982861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-podres\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.983084 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.982957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-lib-modules\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.983084 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.982995 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-podres\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.983084 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.983064 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8a00e7a-440d-48df-af38-51b80509ef4e-lib-modules\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:02:59.991933 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:02:59.991906 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hc8\" (UniqueName: \"kubernetes.io/projected/c8a00e7a-440d-48df-af38-51b80509ef4e-kube-api-access-s7hc8\") pod \"perf-node-gather-daemonset-79rlp\" (UID: \"c8a00e7a-440d-48df-af38-51b80509ef4e\") " pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:03:00.038881 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:00.038857 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:03:00.176751 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:00.176725 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp"] Jul 06 20:03:00.178439 ip-10-0-131-211 kubenswrapper[2571]: W0706 20:03:00.178409 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc8a00e7a_440d_48df_af38_51b80509ef4e.slice/crio-dc2084552888340530ca9c40608f35c8a05d893f3338a6900b2750e72a3d4eca WatchSource:0}: Error finding container dc2084552888340530ca9c40608f35c8a05d893f3338a6900b2750e72a3d4eca: Status 404 returned error can't find the container with id dc2084552888340530ca9c40608f35c8a05d893f3338a6900b2750e72a3d4eca Jul 06 20:03:00.484127 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:00.484066 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" event={"ID":"c8a00e7a-440d-48df-af38-51b80509ef4e","Type":"ContainerStarted","Data":"12e19f1a94d4dde7f710a18bf0d4ef9249dd92740499ef4e4eb52a4ddbd67082"} Jul 06 20:03:00.484354 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:00.484153 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:03:00.484354 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:00.484183 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" event={"ID":"c8a00e7a-440d-48df-af38-51b80509ef4e","Type":"ContainerStarted","Data":"dc2084552888340530ca9c40608f35c8a05d893f3338a6900b2750e72a3d4eca"} Jul 06 20:03:00.505035 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:00.504991 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" podStartSLOduration=1.5049781530000002 podStartE2EDuration="1.504978153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:03:00.503925164 +0000 UTC m=+2567.564272090" watchObservedRunningTime="2026-07-06 20:03:00.504978153 +0000 UTC m=+2567.565325078" Jul 06 20:03:00.831771 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:00.831741 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d4b949455-2qh74_ce416fef-b171-432c-8931-936823a8529b/console/0.log" Jul 06 20:03:02.521773 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:02.521747 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6flwj_21aa5d6c-ede2-462b-b876-c95e1b1b4287/dns/0.log" Jul 06 20:03:02.543373 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:02.543354 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6flwj_21aa5d6c-ede2-462b-b876-c95e1b1b4287/kube-rbac-proxy/0.log" Jul 06 20:03:02.658872 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:02.658849 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cb5fx_fe3d67f2-49f2-4ea8-8fab-29a1801d431a/dns-node-resolver/0.log" Jul 06 20:03:03.326928 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:03.326901 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tbczb_7687a7fd-1cbf-4521-a462-e976d785f5e5/node-ca/0.log" Jul 06 20:03:05.639186 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:05.639158 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5xtc8_3a55cefa-2f6c-45a7-99aa-d2d14317c804/serve-healthcheck-canary/0.log" Jul 06 20:03:06.462161 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:06.462132 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8dsxn_b73892e7-6ba0-486b-ae84-98b711e76ed2/kube-rbac-proxy/0.log" Jul 06 20:03:06.487310 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:06.487285 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8dsxn_b73892e7-6ba0-486b-ae84-98b711e76ed2/exporter/0.log" Jul 06 20:03:06.498993 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:06.498974 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sxl92/perf-node-gather-daemonset-79rlp" Jul 06 20:03:06.521085 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:06.521067 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8dsxn_b73892e7-6ba0-486b-ae84-98b711e76ed2/extractor/0.log" Jul 06 20:03:08.997424 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:08.997365 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2xttg_4694615b-ae46-44cd-8a55-2f1b46d595ee/manager/0.log" Jul 06 20:03:09.167976 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:09.167944 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bq6ll_7d2ac511-e78f-4dfc-adb9-205defb7f9f5/manager/0.log" Jul 06 20:03:09.625905 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:09.625871 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69fd78d8d5-7bxvm_cdf137ce-5324-4116-906d-a8ee2b92c24b/manager/0.log" Jul 06 20:03:09.650015 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:09.649992 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69fd78d8d5-7bxvm_cdf137ce-5324-4116-906d-a8ee2b92c24b/copy-manifests/0.log" Jul 06 20:03:11.262326 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:11.262283 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d885dd8f9-xg5sq_1b9f93d7-7753-451d-b29b-7d3ebcf1b9fa/manager/0.log" Jul 06 20:03:11.292574 ip-10-0-131-211 kubenswrapper[2571]: I0706 20:03:11.292549 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-bp2lz_84bed43d-3933-42f6-b76d-7803a642380e/openshift-lws-operator/0.log"