Jul 16 04:37:29.676414 ip-10-0-140-3 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 04:37:29.676425 ip-10-0-140-3 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 04:37:29.676432 ip-10-0-140-3 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 04:37:29.676633 ip-10-0-140-3 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 04:37:39.858511 ip-10-0-140-3 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 04:37:39.858529 ip-10-0-140-3 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 42983dbc9267414687ce7580b031950b -- Jul 16 04:39:11.302451 ip-10-0-140-3 systemd[1]: Starting Kubernetes Kubelet... Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.754901 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759859 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759873 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759878 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759881 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759887 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759896 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759900 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 04:39:11.781691 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759903 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759906 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759909 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759912 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759915 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759918 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759920 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759923 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759926 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759929 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759931 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759934 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759937 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759939 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759942 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759945 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759949 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759952 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759955 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759957 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 04:39:12.099340 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759960 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759963 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759966 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759968 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759972 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759975 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759977 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759980 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759983 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759986 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759988 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759991 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759994 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.759997 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760000 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760002 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760005 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760008 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760010 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760013 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 04:39:12.111843 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760015 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 04:39:11.834138 ip-10-0-140-3 systemd[1]: Started Kubernetes Kubelet. Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760018 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760021 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760024 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760027 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760031 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760035 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760038 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760041 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760044 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760047 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760049 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760052 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760055 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760057 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760060 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760062 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760105 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760148 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 04:39:12.112724 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760153 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760158 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760163 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760166 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760169 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760184 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760189 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760197 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760203 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760208 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760214 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760219 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.760232 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762065 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762085 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762092 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762098 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762103 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762114 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762119 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 04:39:12.113532 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762141 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762266 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762279 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762284 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762287 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762291 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762294 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762296 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762299 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762303 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762305 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762308 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.762311 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763039 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763046 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763049 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763052 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763055 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763057 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763061 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 04:39:12.114204 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763064 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763067 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763070 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763073 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763076 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763078 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763081 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763084 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763086 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763089 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763092 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763094 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763101 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763104 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763107 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763109 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763112 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763114 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763117 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 04:39:12.114981 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763119 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763122 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763125 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763128 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763130 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763133 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763135 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763141 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763145 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763148 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763150 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763153 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763156 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763158 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763161 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763164 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763166 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763169 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763171 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763196 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 04:39:12.115583 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763200 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763204 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763207 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763210 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763212 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763215 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763218 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763220 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763223 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763225 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763228 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763231 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763233 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763236 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763238 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763241 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763245 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763248 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763250 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763255 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 04:39:12.116231 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763259 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763261 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763264 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763267 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763269 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763272 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763274 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763277 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763281 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763285 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763288 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763290 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763293 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763297 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763300 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763303 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763306 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763309 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763312 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 04:39:12.116821 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763314 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763317 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763319 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763322 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763325 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763329 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763331 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763334 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763336 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763339 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763341 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763344 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763347 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763351 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763765 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763770 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763773 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763777 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763779 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763782 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 04:39:12.117448 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763785 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763788 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763792 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763795 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763798 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763800 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763803 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763806 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763817 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763819 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763822 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763825 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763828 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763833 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763837 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763841 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763847 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763852 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763857 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763862 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 04:39:12.118007 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763865 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763868 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763871 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763873 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763877 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763881 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763884 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763887 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763890 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763892 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763895 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763898 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763900 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763903 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763906 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763908 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763911 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763914 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763916 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763919 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 04:39:12.118634 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763921 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763925 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763928 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763931 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763933 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763936 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763939 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763942 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763944 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763947 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763949 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763952 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763954 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763957 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763959 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763962 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763966 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763970 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763972 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763975 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 04:39:12.119223 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763978 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763980 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763982 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763985 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763988 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763990 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763993 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763996 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.763999 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764001 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764005 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764008 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764010 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764013 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764016 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764018 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764021 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764023 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764026 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 04:39:12.119855 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764028 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764032 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764036 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764039 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764042 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764045 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764047 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764050 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764052 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764057 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764060 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764063 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764066 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764069 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764143 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764150 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764157 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764168 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764185 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764189 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764193 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 04:39:12.120414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764198 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764201 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764204 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764207 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764210 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764214 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764217 2585 flags.go:64] FLAG: --cgroup-root="" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764220 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764223 2585 flags.go:64] FLAG: --client-ca-file="" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764226 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764229 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764234 2585 flags.go:64] FLAG: --cluster-domain="" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764237 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764240 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764243 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764247 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764250 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764253 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764256 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764259 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764264 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764268 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764271 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764274 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 04:39:12.121222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764278 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764282 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764285 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764288 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764291 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764294 2585 flags.go:64] FLAG: --enable-server="true" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764297 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764301 2585 flags.go:64] FLAG: --event-burst="100" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764304 2585 flags.go:64] FLAG: --event-qps="50" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764307 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764310 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764312 2585 flags.go:64] FLAG: --eviction-hard="" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764316 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764319 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764322 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764325 2585 flags.go:64] FLAG: --eviction-soft="" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764329 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764331 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764334 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764337 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764340 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764343 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764346 2585 flags.go:64] FLAG: --feature-gates="" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764349 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764352 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 04:39:12.121855 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764355 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764358 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764361 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764366 2585 flags.go:64] FLAG: --help="false" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764370 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764373 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764376 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764379 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764382 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764386 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764389 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764392 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764395 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764398 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764401 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764404 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764407 2585 flags.go:64] FLAG: --kube-reserved="" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764410 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764413 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764416 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764419 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764422 2585 flags.go:64] FLAG: --lock-file="" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764425 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764428 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 04:39:12.122584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764431 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764437 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764440 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764442 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764445 2585 flags.go:64] FLAG: --logging-format="text" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764448 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764452 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764455 2585 flags.go:64] FLAG: --manifest-url="" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764458 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764462 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764465 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764470 2585 flags.go:64] FLAG: --max-pods="110" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764475 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764478 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764481 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764484 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764487 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764490 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764493 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764500 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764503 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764507 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764510 2585 flags.go:64] FLAG: --pod-cidr="" Jul 16 04:39:12.123344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764512 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764517 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764520 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764523 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764526 2585 flags.go:64] FLAG: --port="10250" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764530 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764533 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c7d551b9fac8b0a1" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764536 2585 flags.go:64] FLAG: --qos-reserved="" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764539 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764542 2585 flags.go:64] FLAG: --register-node="true" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764545 2585 flags.go:64] FLAG: --register-with-taints="" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764549 2585 flags.go:64] FLAG: --registry-burst="10" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764552 2585 flags.go:64] FLAG: --registry-qps="5" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764555 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764558 2585 flags.go:64] FLAG: --reserved-memory="" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764562 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764565 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764568 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764571 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764574 2585 flags.go:64] FLAG: --runonce="false" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764577 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764582 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764585 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764588 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764591 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764594 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 04:39:12.123987 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764597 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764600 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764603 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764606 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764609 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764612 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764615 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764618 2585 flags.go:64] FLAG: --system-cgroups="" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764621 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764627 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764630 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764632 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764637 2585 flags.go:64] FLAG: --tls-min-version="" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764640 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764643 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764645 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764648 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764652 2585 flags.go:64] FLAG: --v="2" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764656 2585 flags.go:64] FLAG: --version="false" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764660 2585 flags.go:64] FLAG: --vmodule="" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764664 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.764668 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764780 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764785 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 04:39:12.125085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764788 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764790 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764795 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764803 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764806 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764809 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764812 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764815 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764817 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764820 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764823 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764825 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764828 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764831 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764833 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764836 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764839 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764841 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764844 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 04:39:12.125931 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764846 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764849 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764852 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764854 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764857 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764860 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764863 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764866 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764868 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764871 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764874 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764876 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764879 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764881 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764884 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764886 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764892 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764895 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764898 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764900 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 04:39:12.288676 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764903 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764906 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764909 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764911 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764914 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764916 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764919 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764922 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764925 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764927 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764930 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764933 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764936 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764938 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764941 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764943 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764946 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764948 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764951 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764955 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 04:39:12.289502 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764959 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764962 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764964 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764967 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764969 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764972 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764974 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764977 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764981 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764984 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764987 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764989 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764992 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764995 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.764997 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765000 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765002 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765005 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765007 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765010 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 04:39:12.290061 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765022 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765025 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765027 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765030 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765033 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765035 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765038 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765041 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765043 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765046 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765048 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765051 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765054 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765057 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765060 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765062 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765065 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 04:39:12.290674 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.765069 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.765784 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.772930 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.772949 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.773820 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.773832 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773921 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773929 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773933 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773938 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773941 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773945 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773949 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773954 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 04:39:12.291123 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773960 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773965 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773970 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773974 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773979 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773983 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773989 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773994 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.773996 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774001 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774006 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774009 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774014 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774018 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774022 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774026 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774030 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774034 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774037 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 04:39:12.291581 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774041 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774045 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774050 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774054 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774059 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774064 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774068 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774072 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774076 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774080 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774084 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774088 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774092 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774096 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774100 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774104 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774109 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774113 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774117 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774121 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774125 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 04:39:12.292085 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774129 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774132 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774136 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774140 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774147 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774152 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774155 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774160 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774163 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774167 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774171 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774189 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774193 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774200 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774210 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774214 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774219 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774223 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774227 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774231 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 04:39:12.292669 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774235 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774239 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774243 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774247 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774251 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774254 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774258 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774262 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774266 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774270 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774275 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774281 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774285 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774289 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774293 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774296 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774302 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774305 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774309 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 04:39:12.293499 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774313 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774317 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774321 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774324 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774327 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774332 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774336 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774340 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774345 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774348 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774352 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774355 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.774362 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774520 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774527 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 04:39:12.294140 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774532 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774535 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774540 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774543 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774547 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774551 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774554 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774558 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774562 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774566 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774570 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774574 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774578 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774582 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774586 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774590 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774594 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774598 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774601 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774605 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 04:39:12.294777 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774609 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774612 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774616 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774620 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774625 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774631 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774637 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774641 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774644 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774649 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774654 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774659 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774663 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774667 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774671 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774675 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774679 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774682 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 04:39:12.295404 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774686 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774690 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774693 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774697 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774701 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774704 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774708 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774712 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774717 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774721 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774725 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774731 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774735 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774739 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774743 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774747 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774750 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774754 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774759 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774763 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 04:39:12.296025 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774767 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774772 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774776 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774779 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774784 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774787 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774791 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774795 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774799 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774803 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774807 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774810 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774814 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774818 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774822 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774825 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774829 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774833 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774838 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774841 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 04:39:12.296641 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774847 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774850 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774853 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774856 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774859 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774863 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774867 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774871 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774875 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774879 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774884 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774888 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774892 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774896 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774901 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774905 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774909 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774912 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 04:39:12.297467 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:11.774917 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.774924 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.775069 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.777501 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.779288 2585 server.go:1019] "Starting client certificate rotation" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.779389 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.779454 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.782114 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.787573 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.794896 2585 log.go:25] "Validated CRI v1 runtime API" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.797427 2585 log.go:25] "Validated CRI v1 image API" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.799469 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 04:39:12.298083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.802921 2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7c1a7fba-ace0-456f-8c31-f766a4b3f467:/dev/nvme0n1p4 a6b7c7d0-6fe8-447e-b252-773a2a97fe6a:/dev/nvme0n1p3] Jul 16 04:39:12.298497 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.802938 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 04:39:12.298701 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.808009 2585 manager.go:216] Machine: {Timestamp:2026-07-16 04:39:11.806567473 +0000 UTC m=+0.371207039 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099282 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22b66ca84424b98ba048d42c6e2175 SystemUUID:ec22b66c-a844-24b9-8ba0-48d42c6e2175 BootID:42983dbc-9267-4146-87ce-7580b031950b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:80:09:34:26:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:80:09:34:26:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:f9:51:6d:28:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 04:39:12.298701 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.808110 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 04:39:12.298701 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.808197 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 04:39:12.298701 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.809342 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 04:39:12.298701 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.811759 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 04:39:12.298701 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.811788 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-3.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 04:39:12.298701 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.813342 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.813353 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.813361 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.814472 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.817043 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.821759 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.821875 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.821901 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.821913 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.821930 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.821941 2585 kubelet.go:398] "Adding apiserver pod source" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.821950 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.822297 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.822313 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.824505 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827087 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827114 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827290 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827303 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827309 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827315 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827323 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827329 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827984 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827991 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.827998 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.828004 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.828013 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.828021 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.828915 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.828922 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.833464 2585 server.go:1268] "Started kubelet" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.833570 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.833580 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.833654 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.837664 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.837795 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-3.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.838100 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.838375 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.839518 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 04:39:12.298951 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.840459 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.842665 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vch6g" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.843974 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-3.ec2.internal.18c2ab4d858895d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-3.ec2.internal,UID:ip-10-0-140-3.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-3.ec2.internal,},FirstTimestamp:2026-07-16 04:39:11.833290196 +0000 UTC m=+0.397929745,LastTimestamp:2026-07-16 04:39:11.833290196 +0000 UTC m=+0.397929745,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-3.ec2.internal,}" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.847070 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.847078 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.847170 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.847804 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.847812 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.847819 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.847861 2585 factory.go:55] Registering systemd factory Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.847873 2585 factory.go:223] Registration of the systemd container factory successfully Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.847901 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.847913 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.848044 2585 factory.go:153] Registering CRI-O factory Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.848055 2585 factory.go:223] Registration of the crio container factory successfully Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.848105 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.848125 2585 factory.go:103] Registering Raw factory Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.848141 2585 manager.go:1196] Started watching for new ooms in manager Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.848493 2585 manager.go:349] Starting recovery of all containers Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.850882 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vch6g" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.852644 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.853072 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.856096 2585 manager.go:354] Recovery completed Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.860715 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.862301 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.862325 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.862334 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.862806 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.862814 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.862815 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.862839 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.863487 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.863512 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 04:39:12.300016 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.863522 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.864884 2585 policy_none.go:49] "None policy: Start" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.864896 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.864906 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.866908 2585 policy_none.go:47] "Start" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.948038 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.955411 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.956599 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.956638 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.956661 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.956700 2585 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.958707 2585 manager.go:337] "Starting Device Plugin manager" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.958742 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.958756 2585 server.go:87] "Starting device plugin registration server" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.959073 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.959086 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.959403 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.959477 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.959486 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.959764 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:11.959839 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:11.960504 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.057273 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal"] Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.057335 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.058406 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.058434 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.058447 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.059466 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.060133 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.060155 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.060166 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.060199 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.060801 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.060912 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.060938 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.061430 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.061457 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.061471 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID" Jul 16 04:39:12.301187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.061504 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.061523 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.061532 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:12.062654 2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-3.ec2.internal\" not found" node="ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.064117 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.064138 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.064817 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientMemory" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.064844 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.064856 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeHasSufficientPID" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.084045 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:12.084063 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-3.ec2.internal\": node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:12.100115 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-3.ec2.internal\" not found" node="ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:12.104514 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-3.ec2.internal\" not found" node="ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:12.131521 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.149208 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e834723c39bbc8029a7518c929607dfd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"e834723c39bbc8029a7518c929607dfd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.149230 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e834723c39bbc8029a7518c929607dfd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"e834723c39bbc8029a7518c929607dfd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.149252 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2d100063295820fbe31810c8234ecb8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-3.ec2.internal\" (UID: \"b2d100063295820fbe31810c8234ecb8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:12.232041 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.249549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e834723c39bbc8029a7518c929607dfd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"e834723c39bbc8029a7518c929607dfd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.249573 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e834723c39bbc8029a7518c929607dfd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"e834723c39bbc8029a7518c929607dfd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.249590 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2d100063295820fbe31810c8234ecb8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-3.ec2.internal\" (UID: \"b2d100063295820fbe31810c8234ecb8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.302248 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.249628 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2d100063295820fbe31810c8234ecb8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-3.ec2.internal\" (UID: \"b2d100063295820fbe31810c8234ecb8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.444062 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.249653 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e834723c39bbc8029a7518c929607dfd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"e834723c39bbc8029a7518c929607dfd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.444062 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.249656 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e834723c39bbc8029a7518c929607dfd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal\" (UID: \"e834723c39bbc8029a7518c929607dfd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.444062 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:12.332938 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:12.444062 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.402941 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.444062 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:12.407082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:12.444062 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:12.434029 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:13.834482 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:12.534625 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:13.954669 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:13.881802 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-3.ec2.internal\" not found" Jul 16 04:39:13.954669 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:13.925200 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 04:39:14.069305 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:13.998601 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" Jul 16 04:39:14.069305 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.015924 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 04:39:14.069305 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.017300 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:14.069305 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.024488 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 04:39:14.069305 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.024606 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 04:39:14.069305 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.024642 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 04:39:14.069305 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.024674 2585 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a162f57d183f648a0847f28e16979aed-4f7be0ade2dab6eb.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.140.3:58718->52.21.221.10:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:14.069305 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.068799 2585 apiserver.go:52] "Watching apiserver" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.084301 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.085103 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-cgjjz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal","openshift-multus/multus-4wmg9","openshift-multus/multus-additional-cni-plugins-55n8f","openshift-multus/network-metrics-daemon-nsrxl","openshift-network-diagnostics/network-check-target-lgzrj","kube-system/konnectivity-agent-nnfvz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c","openshift-dns/node-resolver-tqh7x","openshift-network-operator/iptables-alerter-7887n","openshift-ovn-kubernetes/ovnkube-node-s27vv","kube-system/global-pull-secret-syncer-l6277","openshift-cluster-node-tuning-operator/tuned-szbxb"] Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.087273 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.089120 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.089484 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t7bbd\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.091847 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.091857 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.091976 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.092060 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.092299 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.092610 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.093178 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lcn5n\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.093376 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.093885 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.095145 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-62bdd\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.095286 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.095459 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.095508 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.095737 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.097613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.097650 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-os-release\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.161554 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098853 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-run-netns\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-hostroot\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098880 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48c4ca63-e0a4-4cb0-be52-2476b3758d98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098895 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-os-release\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098909 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098928 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7dsl\" (UniqueName: \"kubernetes.io/projected/42f5e13d-20f3-46ed-ab64-13ce54e5e719-kube-api-access-k7dsl\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098950 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/496d22a4-290a-4a92-9a34-b3d48db8a114-agent-certs\") pod \"konnectivity-agent-nnfvz\" (UID: \"496d22a4-290a-4a92-9a34-b3d48db8a114\") " pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-cni-dir\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098980 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-run-k8s-cni-cncf-io\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.098994 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-daemon-config\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-run-multus-certs\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099021 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-system-cni-dir\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099035 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00a467c5-6e94-4450-b9f0-c84d4e1de57c-cni-binary-copy\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099049 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-socket-dir-parent\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.162484 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099065 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-var-lib-cni-multus\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-cnibin\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099101 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r928l\" (UniqueName: \"kubernetes.io/projected/48c4ca63-e0a4-4cb0-be52-2476b3758d98-kube-api-access-r928l\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099120 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099136 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/496d22a4-290a-4a92-9a34-b3d48db8a114-konnectivity-ca\") pod \"konnectivity-agent-nnfvz\" (UID: \"496d22a4-290a-4a92-9a34-b3d48db8a114\") " pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099161 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-cnibin\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099174 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-var-lib-kubelet\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099189 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48c4ca63-e0a4-4cb0-be52-2476b3758d98-cni-binary-copy\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099203 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48c4ca63-e0a4-4cb0-be52-2476b3758d98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099217 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099232 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-etc-kubernetes\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099246 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-system-cni-dir\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099263 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-var-lib-cni-bin\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099276 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-conf-dir\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.163110 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099290 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc65s\" (UniqueName: \"kubernetes.io/projected/00a467c5-6e94-4450-b9f0-c84d4e1de57c-kube-api-access-dc65s\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.099496 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.101397 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.102025 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 04:34:13 +0000 UTC" deadline="2028-03-28 21:24:05.029235184 +0000 UTC" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.102057 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14920h44m50.927179867s" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.102194 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.102363 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.102484 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45lbx\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.102628 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.103624 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.103810 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s2s5z\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.103937 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.104090 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.104310 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.106081 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.106619 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.106834 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.106967 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v8sr4\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.108557 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w926s\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.108745 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.109000 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.109054 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.109225 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.111960 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.112414 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.112461 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hd96j\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.112472 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.112505 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.113450 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 04:39:14.163741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.113470 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 04:39:14.164584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.113516 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 04:39:14.164584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.113522 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 04:39:14.164584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.114671 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.164584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.116083 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 04:39:14.164584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.117360 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dltjz\"" Jul 16 04:39:14.164584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.117422 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 04:39:14.164584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.117502 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 04:39:14.164584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.143397 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vxf6j" Jul 16 04:39:14.164584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.153188 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vxf6j" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.194901 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199754 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199789 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-run-netns\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-hostroot\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199829 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199876 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-hostroot\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199880 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-lib-modules\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199912 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1258efcc-2979-4c2a-a9a3-26f06d1699bd-ovn-node-metrics-cert\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199941 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dsl\" (UniqueName: \"kubernetes.io/projected/42f5e13d-20f3-46ed-ab64-13ce54e5e719-kube-api-access-k7dsl\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199962 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-device-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.199992 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-run\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200011 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-host\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200032 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-run-systemd\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200098 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-etc-openvswitch\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.254310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-cni-netd\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200139 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-systemd\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200161 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-run-multus-certs\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200175 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fvs\" (UniqueName: \"kubernetes.io/projected/69a01906-2a04-452a-b44e-9f6971572c5d-kube-api-access-w8fvs\") pod \"node-ca-cgjjz\" (UID: \"69a01906-2a04-452a-b44e-9f6971572c5d\") " pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200193 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-system-cni-dir\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200209 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-run-multus-certs\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-system-cni-dir\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200230 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-var-lib-cni-multus\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-var-lib-cni-multus\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200270 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-sys-fs\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200299 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n28n\" (UniqueName: \"kubernetes.io/projected/3680ab79-d032-4fac-8abe-f125045074cd-kube-api-access-2n28n\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200326 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glphh\" (UniqueName: \"kubernetes.io/projected/1258efcc-2979-4c2a-a9a3-26f06d1699bd-kube-api-access-glphh\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200351 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-cnibin\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200375 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:14.257082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200398 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/69a01906-2a04-452a-b44e-9f6971572c5d-host\") pod \"node-ca-cgjjz\" (UID: \"69a01906-2a04-452a-b44e-9f6971572c5d\") " pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200419 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/69a01906-2a04-452a-b44e-9f6971572c5d-serviceca\") pod \"node-ca-cgjjz\" (UID: \"69a01906-2a04-452a-b44e-9f6971572c5d\") " pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200442 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-var-lib-openvswitch\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200448 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-cnibin\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200465 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/496d22a4-290a-4a92-9a34-b3d48db8a114-konnectivity-ca\") pod \"konnectivity-agent-nnfvz\" (UID: \"496d22a4-290a-4a92-9a34-b3d48db8a114\") " pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.200478 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-var-lib-kubelet\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48c4ca63-e0a4-4cb0-be52-2476b3758d98-cni-binary-copy\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.200542 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs podName:42f5e13d-20f3-46ed-ab64-13ce54e5e719 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:14.70052663 +0000 UTC m=+2.018575166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs") pod "network-metrics-daemon-nsrxl" (UID: "42f5e13d-20f3-46ed-ab64-13ce54e5e719") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200552 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-var-lib-kubelet\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200559 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48c4ca63-e0a4-4cb0-be52-2476b3758d98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200596 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e63d7c4-a323-4925-a1e2-06690436efe2-tmp-dir\") pod \"node-resolver-tqh7x\" (UID: \"0e63d7c4-a323-4925-a1e2-06690436efe2\") " pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200621 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-sysconfig\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200712 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3680ab79-d032-4fac-8abe-f125045074cd-etc-tuned\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.257724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200740 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-run-ovn\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200813 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-etc-kubernetes\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200835 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-sysctl-conf\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200875 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-etc-kubernetes\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200898 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3dd29329-28ee-4067-beda-dc5dc41c707a-iptables-alerter-script\") pod \"iptables-alerter-7887n\" (UID: \"3dd29329-28ee-4067-beda-dc5dc41c707a\") " pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200914 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-cni-bin\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200928 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1258efcc-2979-4c2a-a9a3-26f06d1699bd-ovnkube-script-lib\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200951 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-var-lib-cni-bin\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200985 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgjg\" (UniqueName: \"kubernetes.io/projected/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-kube-api-access-tcgjg\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200989 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-var-lib-cni-bin\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.200999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-log-socket\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201018 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-os-release\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.259276 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201044 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-run-netns\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201058 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48c4ca63-e0a4-4cb0-be52-2476b3758d98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201072 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3680ab79-d032-4fac-8abe-f125045074cd-tmp\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201087 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49qg\" (UniqueName: \"kubernetes.io/projected/3dd29329-28ee-4067-beda-dc5dc41c707a-kube-api-access-d49qg\") pod \"iptables-alerter-7887n\" (UID: \"3dd29329-28ee-4067-beda-dc5dc41c707a\") " pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201107 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-run-openvswitch\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201117 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-run-netns\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201124 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-node-log\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201162 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-os-release\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201174 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-os-release\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201202 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-socket-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201217 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-kubelet\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201217 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-os-release\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201231 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-slash\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.260142 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201263 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201285 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/496d22a4-290a-4a92-9a34-b3d48db8a114-agent-certs\") pod \"konnectivity-agent-nnfvz\" (UID: \"496d22a4-290a-4a92-9a34-b3d48db8a114\") " pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-cni-dir\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201327 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-run-k8s-cni-cncf-io\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201354 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-daemon-config\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201377 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqwn6\" (UniqueName: \"kubernetes.io/projected/0e63d7c4-a323-4925-a1e2-06690436efe2-kube-api-access-sqwn6\") pod \"node-resolver-tqh7x\" (UID: \"0e63d7c4-a323-4925-a1e2-06690436efe2\") " pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201417 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-modprobe-d\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201422 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-host-run-k8s-cni-cncf-io\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201442 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201464 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-cni-dir\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201470 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-sys\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201497 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00a467c5-6e94-4450-b9f0-c84d4e1de57c-cni-binary-copy\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-socket-dir-parent\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201642 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-socket-dir-parent\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.261303 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201655 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2385ede2-8c97-424e-b560-19011a292b9e-kubelet-config\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201683 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-sysctl-d\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201717 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r928l\" (UniqueName: \"kubernetes.io/projected/48c4ca63-e0a4-4cb0-be52-2476b3758d98-kube-api-access-r928l\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201747 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e63d7c4-a323-4925-a1e2-06690436efe2-hosts-file\") pod \"node-resolver-tqh7x\" (UID: \"0e63d7c4-a323-4925-a1e2-06690436efe2\") " pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201822 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-var-lib-kubelet\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-cnibin\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201913 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-cnibin\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201940 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-systemd-units\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1258efcc-2979-4c2a-a9a3-26f06d1699bd-ovnkube-config\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.201992 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1258efcc-2979-4c2a-a9a3-26f06d1699bd-env-overrides\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202038 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-registration-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2385ede2-8c97-424e-b560-19011a292b9e-dbus\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202114 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-kubernetes\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202142 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dd29329-28ee-4067-beda-dc5dc41c707a-host-slash\") pod \"iptables-alerter-7887n\" (UID: \"3dd29329-28ee-4067-beda-dc5dc41c707a\") " pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.262184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202155 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/496d22a4-290a-4a92-9a34-b3d48db8a114-konnectivity-ca\") pod \"konnectivity-agent-nnfvz\" (UID: \"496d22a4-290a-4a92-9a34-b3d48db8a114\") " pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202172 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-system-cni-dir\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202202 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-conf-dir\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202233 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc65s\" (UniqueName: \"kubernetes.io/projected/00a467c5-6e94-4450-b9f0-c84d4e1de57c-kube-api-access-dc65s\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202280 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-system-cni-dir\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202329 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-conf-dir\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202586 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48c4ca63-e0a4-4cb0-be52-2476b3758d98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202684 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48c4ca63-e0a4-4cb0-be52-2476b3758d98-cni-binary-copy\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202706 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48c4ca63-e0a4-4cb0-be52-2476b3758d98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.202707 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.203127 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48c4ca63-e0a4-4cb0-be52-2476b3758d98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.204779 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00a467c5-6e94-4450-b9f0-c84d4e1de57c-multus-daemon-config\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.205195 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00a467c5-6e94-4450-b9f0-c84d4e1de57c-cni-binary-copy\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.206410 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/496d22a4-290a-4a92-9a34-b3d48db8a114-agent-certs\") pod \"konnectivity-agent-nnfvz\" (UID: \"496d22a4-290a-4a92-9a34-b3d48db8a114\") " pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.212454 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dsl\" (UniqueName: \"kubernetes.io/projected/42f5e13d-20f3-46ed-ab64-13ce54e5e719-kube-api-access-k7dsl\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:14.266671 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.217151 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r928l\" (UniqueName: \"kubernetes.io/projected/48c4ca63-e0a4-4cb0-be52-2476b3758d98-kube-api-access-r928l\") pod \"multus-additional-cni-plugins-55n8f\" (UID: \"48c4ca63-e0a4-4cb0-be52-2476b3758d98\") " pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.267512 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.217278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc65s\" (UniqueName: \"kubernetes.io/projected/00a467c5-6e94-4450-b9f0-c84d4e1de57c-kube-api-access-dc65s\") pod \"multus-4wmg9\" (UID: \"00a467c5-6e94-4450-b9f0-c84d4e1de57c\") " pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.267512 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.217789 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 04:39:14.267512 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.217804 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 04:39:14.267512 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.217812 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w69nk for pod openshift-network-diagnostics/network-check-target-lgzrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:14.267512 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.217855 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk podName:255f6da7-bab4-4b7c-984f-cd1ddda313e9 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:14.717842504 +0000 UTC m=+2.035891061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w69nk" (UniqueName: "kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk") pod "network-check-target-lgzrj" (UID: "255f6da7-bab4-4b7c-984f-cd1ddda313e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302550 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgjg\" (UniqueName: \"kubernetes.io/projected/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-kube-api-access-tcgjg\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302565 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-log-socket\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302599 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3680ab79-d032-4fac-8abe-f125045074cd-tmp\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302634 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49qg\" (UniqueName: \"kubernetes.io/projected/3dd29329-28ee-4067-beda-dc5dc41c707a-kube-api-access-d49qg\") pod \"iptables-alerter-7887n\" (UID: \"3dd29329-28ee-4067-beda-dc5dc41c707a\") " pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302649 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-log-socket\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-run-openvswitch\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302670 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302681 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-node-log\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302717 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-run-openvswitch\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302720 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-node-log\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302734 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-socket-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-kubelet\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.373649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302782 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-slash\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302806 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302830 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-kubelet\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302851 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqwn6\" (UniqueName: \"kubernetes.io/projected/0e63d7c4-a323-4925-a1e2-06690436efe2-kube-api-access-sqwn6\") pod \"node-resolver-tqh7x\" (UID: \"0e63d7c4-a323-4925-a1e2-06690436efe2\") " pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302876 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-modprobe-d\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302879 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302875 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-socket-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.302974 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-modprobe-d\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303008 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-slash\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303038 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-sys\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303109 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2385ede2-8c97-424e-b560-19011a292b9e-kubelet-config\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303134 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-sysctl-d\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303157 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2385ede2-8c97-424e-b560-19011a292b9e-kubelet-config\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.374726 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303165 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303176 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e63d7c4-a323-4925-a1e2-06690436efe2-hosts-file\") pod \"node-resolver-tqh7x\" (UID: \"0e63d7c4-a323-4925-a1e2-06690436efe2\") " pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303202 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-var-lib-kubelet\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303224 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-systemd-units\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e63d7c4-a323-4925-a1e2-06690436efe2-hosts-file\") pod \"node-resolver-tqh7x\" (UID: \"0e63d7c4-a323-4925-a1e2-06690436efe2\") " pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303244 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1258efcc-2979-4c2a-a9a3-26f06d1699bd-ovnkube-config\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303266 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1258efcc-2979-4c2a-a9a3-26f06d1699bd-env-overrides\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303274 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-sysctl-d\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303283 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-var-lib-kubelet\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-registration-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303283 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-systemd-units\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303316 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2385ede2-8c97-424e-b560-19011a292b9e-dbus\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303341 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-kubernetes\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303359 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-registration-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303364 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dd29329-28ee-4067-beda-dc5dc41c707a-host-slash\") pod \"iptables-alerter-7887n\" (UID: \"3dd29329-28ee-4067-beda-dc5dc41c707a\") " pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.375411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303408 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-kubernetes\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303416 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303422 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2385ede2-8c97-424e-b560-19011a292b9e-dbus\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303443 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-run-netns\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303135 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-sys\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303468 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dd29329-28ee-4067-beda-dc5dc41c707a-host-slash\") pod \"iptables-alerter-7887n\" (UID: \"3dd29329-28ee-4067-beda-dc5dc41c707a\") " pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303469 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-lib-modules\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303526 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1258efcc-2979-4c2a-a9a3-26f06d1699bd-ovn-node-metrics-cert\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303555 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-device-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303566 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303580 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-run\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303598 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-lib-modules\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303502 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-run-netns\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.303530 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:14.376336 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303683 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-device-dir\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.303697 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret podName:2385ede2-8c97-424e-b560-19011a292b9e nodeName:}" failed. No retries permitted until 2026-07-16 04:39:14.803677962 +0000 UTC m=+2.121726502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret") pod "global-pull-secret-syncer-l6277" (UID: "2385ede2-8c97-424e-b560-19011a292b9e") : object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303643 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-run\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303803 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-host\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303848 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-run-systemd\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303875 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-host\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303887 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-etc-openvswitch\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303914 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-cni-netd\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303915 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-run-systemd\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303941 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-systemd\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303953 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-etc-openvswitch\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303975 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fvs\" (UniqueName: \"kubernetes.io/projected/69a01906-2a04-452a-b44e-9f6971572c5d-kube-api-access-w8fvs\") pod \"node-ca-cgjjz\" (UID: \"69a01906-2a04-452a-b44e-9f6971572c5d\") " pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.303976 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-cni-netd\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.304001 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-systemd\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.376850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.304090 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-sys-fs\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.304115 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n28n\" (UniqueName: \"kubernetes.io/projected/3680ab79-d032-4fac-8abe-f125045074cd-kube-api-access-2n28n\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.304190 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-sys-fs\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.305226 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3680ab79-d032-4fac-8abe-f125045074cd-tmp\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.312932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glphh\" (UniqueName: \"kubernetes.io/projected/1258efcc-2979-4c2a-a9a3-26f06d1699bd-kube-api-access-glphh\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.312985 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/69a01906-2a04-452a-b44e-9f6971572c5d-host\") pod \"node-ca-cgjjz\" (UID: \"69a01906-2a04-452a-b44e-9f6971572c5d\") " pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313008 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/69a01906-2a04-452a-b44e-9f6971572c5d-serviceca\") pod \"node-ca-cgjjz\" (UID: \"69a01906-2a04-452a-b44e-9f6971572c5d\") " pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313033 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-var-lib-openvswitch\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313098 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/69a01906-2a04-452a-b44e-9f6971572c5d-host\") pod \"node-ca-cgjjz\" (UID: \"69a01906-2a04-452a-b44e-9f6971572c5d\") " pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313145 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1258efcc-2979-4c2a-a9a3-26f06d1699bd-env-overrides\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313153 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-var-lib-openvswitch\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313182 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e63d7c4-a323-4925-a1e2-06690436efe2-tmp-dir\") pod \"node-resolver-tqh7x\" (UID: \"0e63d7c4-a323-4925-a1e2-06690436efe2\") " pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-sysconfig\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313244 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3680ab79-d032-4fac-8abe-f125045074cd-etc-tuned\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313260 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-run-ovn\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.377499 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313305 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-sysctl-conf\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313331 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1258efcc-2979-4c2a-a9a3-26f06d1699bd-ovnkube-config\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313334 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3dd29329-28ee-4067-beda-dc5dc41c707a-iptables-alerter-script\") pod \"iptables-alerter-7887n\" (UID: \"3dd29329-28ee-4067-beda-dc5dc41c707a\") " pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313396 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-sysconfig\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313405 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-cni-bin\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313440 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-host-cni-bin\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313454 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/69a01906-2a04-452a-b44e-9f6971572c5d-serviceca\") pod \"node-ca-cgjjz\" (UID: \"69a01906-2a04-452a-b44e-9f6971572c5d\") " pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313466 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1258efcc-2979-4c2a-a9a3-26f06d1699bd-run-ovn\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1258efcc-2979-4c2a-a9a3-26f06d1699bd-ovnkube-script-lib\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313504 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3680ab79-d032-4fac-8abe-f125045074cd-etc-sysctl-conf\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313521 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e63d7c4-a323-4925-a1e2-06690436efe2-tmp-dir\") pod \"node-resolver-tqh7x\" (UID: \"0e63d7c4-a323-4925-a1e2-06690436efe2\") " pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.313896 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1258efcc-2979-4c2a-a9a3-26f06d1699bd-ovnkube-script-lib\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.314411 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3dd29329-28ee-4067-beda-dc5dc41c707a-iptables-alerter-script\") pod \"iptables-alerter-7887n\" (UID: \"3dd29329-28ee-4067-beda-dc5dc41c707a\") " pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.315149 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3680ab79-d032-4fac-8abe-f125045074cd-etc-tuned\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.315223 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1258efcc-2979-4c2a-a9a3-26f06d1699bd-ovn-node-metrics-cert\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.378071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.344169 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqwn6\" (UniqueName: \"kubernetes.io/projected/0e63d7c4-a323-4925-a1e2-06690436efe2-kube-api-access-sqwn6\") pod \"node-resolver-tqh7x\" (UID: \"0e63d7c4-a323-4925-a1e2-06690436efe2\") " pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.378642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.344185 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49qg\" (UniqueName: \"kubernetes.io/projected/3dd29329-28ee-4067-beda-dc5dc41c707a-kube-api-access-d49qg\") pod \"iptables-alerter-7887n\" (UID: \"3dd29329-28ee-4067-beda-dc5dc41c707a\") " pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.378642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.347916 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n28n\" (UniqueName: \"kubernetes.io/projected/3680ab79-d032-4fac-8abe-f125045074cd-kube-api-access-2n28n\") pod \"tuned-szbxb\" (UID: \"3680ab79-d032-4fac-8abe-f125045074cd\") " pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.378642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.349722 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fvs\" (UniqueName: \"kubernetes.io/projected/69a01906-2a04-452a-b44e-9f6971572c5d-kube-api-access-w8fvs\") pod \"node-ca-cgjjz\" (UID: \"69a01906-2a04-452a-b44e-9f6971572c5d\") " pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.378642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.350078 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glphh\" (UniqueName: \"kubernetes.io/projected/1258efcc-2979-4c2a-a9a3-26f06d1699bd-kube-api-access-glphh\") pod \"ovnkube-node-s27vv\" (UID: \"1258efcc-2979-4c2a-a9a3-26f06d1699bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.378642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.351733 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgjg\" (UniqueName: \"kubernetes.io/projected/0a8966ce-dc73-4a6d-94cb-f2430c7dddd3-kube-api-access-tcgjg\") pod \"aws-ebs-csi-driver-node-prw5c\" (UID: \"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.378642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.356101 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 04:39:14.378642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.360580 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:14.378642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.365389 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-szbxb" Jul 16 04:39:14.502686 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.395545 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3680ab79_d032_4fac_8abe_f125045074cd.slice/crio-f5d1bc13600c7d321da0a8c63db56bf16796874de1472db4dfb531a536e363d7 WatchSource:0}: Error finding container f5d1bc13600c7d321da0a8c63db56bf16796874de1472db4dfb531a536e363d7: Status 404 returned error can't find the container with id f5d1bc13600c7d321da0a8c63db56bf16796874de1472db4dfb531a536e363d7 Jul 16 04:39:14.502686 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.396219 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1258efcc_2979_4c2a_a9a3_26f06d1699bd.slice/crio-0e33b8d64bed6b4e9019a375cd423c81462e34c32c27847f54eb3ce56ff89f7d WatchSource:0}: Error finding container 0e33b8d64bed6b4e9019a375cd423c81462e34c32c27847f54eb3ce56ff89f7d: Status 404 returned error can't find the container with id 0e33b8d64bed6b4e9019a375cd423c81462e34c32c27847f54eb3ce56ff89f7d Jul 16 04:39:14.502686 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.397100 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode834723c39bbc8029a7518c929607dfd.slice/crio-913b47ad9e9a418984c1dc27efa7a79256aede4debfefc8bf70dda85fe080edb WatchSource:0}: Error finding container 913b47ad9e9a418984c1dc27efa7a79256aede4debfefc8bf70dda85fe080edb: Status 404 returned error can't find the container with id 913b47ad9e9a418984c1dc27efa7a79256aede4debfefc8bf70dda85fe080edb Jul 16 04:39:14.502686 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.397687 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d100063295820fbe31810c8234ecb8.slice/crio-37a63e386ab0658363d9c77a2b2fea51212d3ae612f95939ba06defaf0bbc605 WatchSource:0}: Error finding container 37a63e386ab0658363d9c77a2b2fea51212d3ae612f95939ba06defaf0bbc605: Status 404 returned error can't find the container with id 37a63e386ab0658363d9c77a2b2fea51212d3ae612f95939ba06defaf0bbc605 Jul 16 04:39:14.502686 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.401465 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 04:39:14.502686 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.429178 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:14.502686 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.439328 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496d22a4_290a_4a92_9a34_b3d48db8a114.slice/crio-3291bc554a2294ca6c45c396c14441e016050cd8aef6672c9b6220fb09b4b912 WatchSource:0}: Error finding container 3291bc554a2294ca6c45c396c14441e016050cd8aef6672c9b6220fb09b4b912: Status 404 returned error can't find the container with id 3291bc554a2294ca6c45c396c14441e016050cd8aef6672c9b6220fb09b4b912 Jul 16 04:39:14.502686 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.478455 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 04:39:14.505615 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.505488 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4wmg9" Jul 16 04:39:14.616545 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.516135 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-55n8f" Jul 16 04:39:14.616545 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.523639 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c4ca63_e0a4_4cb0_be52_2476b3758d98.slice/crio-c0032c28852b7c0cba13ce8e5229d96130cbb45dde19788e081cb98664d627c4 WatchSource:0}: Error finding container c0032c28852b7c0cba13ce8e5229d96130cbb45dde19788e081cb98664d627c4: Status 404 returned error can't find the container with id c0032c28852b7c0cba13ce8e5229d96130cbb45dde19788e081cb98664d627c4 Jul 16 04:39:14.616545 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.539058 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cgjjz" Jul 16 04:39:14.616545 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.547182 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a01906_2a04_452a_b44e_9f6971572c5d.slice/crio-7d4009f66c0add92486687005c37624a309187561cfc4d07d2322f38369e8fec WatchSource:0}: Error finding container 7d4009f66c0add92486687005c37624a309187561cfc4d07d2322f38369e8fec: Status 404 returned error can't find the container with id 7d4009f66c0add92486687005c37624a309187561cfc4d07d2322f38369e8fec Jul 16 04:39:14.616545 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.586699 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" Jul 16 04:39:14.616545 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.595052 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8966ce_dc73_4a6d_94cb_f2430c7dddd3.slice/crio-dff62e6fc3f62198f50d6aab8377971e76292455f1cde1b5d0286eee7da8a358 WatchSource:0}: Error finding container dff62e6fc3f62198f50d6aab8377971e76292455f1cde1b5d0286eee7da8a358: Status 404 returned error can't find the container with id dff62e6fc3f62198f50d6aab8377971e76292455f1cde1b5d0286eee7da8a358 Jul 16 04:39:14.616545 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.603361 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tqh7x" Jul 16 04:39:14.616545 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.612717 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e63d7c4_a323_4925_a1e2_06690436efe2.slice/crio-8fad27ea13d8c0d272e8be13377f3c63093859740258478005acdc7addd8c31f WatchSource:0}: Error finding container 8fad27ea13d8c0d272e8be13377f3c63093859740258478005acdc7addd8c31f: Status 404 returned error can't find the container with id 8fad27ea13d8c0d272e8be13377f3c63093859740258478005acdc7addd8c31f Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.633799 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7887n" Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:14.641336 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd29329_28ee_4067_beda_dc5dc41c707a.slice/crio-bebe5801c6a5db56438626a8c280645ca63087d3ec698751aafdb1f3e8b033bf WatchSource:0}: Error finding container bebe5801c6a5db56438626a8c280645ca63087d3ec698751aafdb1f3e8b033bf: Status 404 returned error can't find the container with id bebe5801c6a5db56438626a8c280645ca63087d3ec698751aafdb1f3e8b033bf Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.717980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.718015 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.718129 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.718143 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.718139 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.718152 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w69nk for pod openshift-network-diagnostics/network-check-target-lgzrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.718215 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs podName:42f5e13d-20f3-46ed-ab64-13ce54e5e719 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:15.718193323 +0000 UTC m=+3.036241860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs") pod "network-metrics-daemon-nsrxl" (UID: "42f5e13d-20f3-46ed-ab64-13ce54e5e719") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:14.741564 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.718259 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk podName:255f6da7-bab4-4b7c-984f-cd1ddda313e9 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:15.718243672 +0000 UTC m=+3.036292210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w69nk" (UniqueName: "kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk") pod "network-check-target-lgzrj" (UID: "255f6da7-bab4-4b7c-984f-cd1ddda313e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:14.847152 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:14.819132 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:14.847152 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.819276 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:14.847152 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:14.819349 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret podName:2385ede2-8c97-424e-b560-19011a292b9e nodeName:}" failed. No retries permitted until 2026-07-16 04:39:15.819331054 +0000 UTC m=+3.137379604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret") pod "global-pull-secret-syncer-l6277" (UID: "2385ede2-8c97-424e-b560-19011a292b9e") : object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:15.156785 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.156051 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 04:34:14 +0000 UTC" deadline="2028-03-17 07:52:23.47189859 +0000 UTC" Jul 16 04:39:15.156785 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.156112 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14643h13m8.315790683s" Jul 16 04:39:15.204166 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.203661 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:15.204166 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:15.203824 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:15.238877 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.238759 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nnfvz" event={"ID":"496d22a4-290a-4a92-9a34-b3d48db8a114","Type":"ContainerStarted","Data":"3291bc554a2294ca6c45c396c14441e016050cd8aef6672c9b6220fb09b4b912"} Jul 16 04:39:15.247651 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.247599 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cgjjz" event={"ID":"69a01906-2a04-452a-b44e-9f6971572c5d","Type":"ContainerStarted","Data":"7d4009f66c0add92486687005c37624a309187561cfc4d07d2322f38369e8fec"} Jul 16 04:39:15.257193 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.257119 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" event={"ID":"e834723c39bbc8029a7518c929607dfd","Type":"ContainerStarted","Data":"913b47ad9e9a418984c1dc27efa7a79256aede4debfefc8bf70dda85fe080edb"} Jul 16 04:39:15.271460 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.271426 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" event={"ID":"b2d100063295820fbe31810c8234ecb8","Type":"ContainerStarted","Data":"37a63e386ab0658363d9c77a2b2fea51212d3ae612f95939ba06defaf0bbc605"} Jul 16 04:39:15.281425 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.281383 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerStarted","Data":"0e33b8d64bed6b4e9019a375cd423c81462e34c32c27847f54eb3ce56ff89f7d"} Jul 16 04:39:15.290898 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.290869 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-szbxb" event={"ID":"3680ab79-d032-4fac-8abe-f125045074cd","Type":"ContainerStarted","Data":"f5d1bc13600c7d321da0a8c63db56bf16796874de1472db4dfb531a536e363d7"} Jul 16 04:39:15.314753 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.313296 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7887n" event={"ID":"3dd29329-28ee-4067-beda-dc5dc41c707a","Type":"ContainerStarted","Data":"bebe5801c6a5db56438626a8c280645ca63087d3ec698751aafdb1f3e8b033bf"} Jul 16 04:39:15.328449 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.328414 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tqh7x" event={"ID":"0e63d7c4-a323-4925-a1e2-06690436efe2","Type":"ContainerStarted","Data":"8fad27ea13d8c0d272e8be13377f3c63093859740258478005acdc7addd8c31f"} Jul 16 04:39:15.333598 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.333560 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" event={"ID":"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3","Type":"ContainerStarted","Data":"dff62e6fc3f62198f50d6aab8377971e76292455f1cde1b5d0286eee7da8a358"} Jul 16 04:39:15.344461 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.344437 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55n8f" event={"ID":"48c4ca63-e0a4-4cb0-be52-2476b3758d98","Type":"ContainerStarted","Data":"c0032c28852b7c0cba13ce8e5229d96130cbb45dde19788e081cb98664d627c4"} Jul 16 04:39:15.355655 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.355627 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4wmg9" event={"ID":"00a467c5-6e94-4450-b9f0-c84d4e1de57c","Type":"ContainerStarted","Data":"6ef3b2e850487edbab305bdf52f96c96ef669ad78ae3de266e660e671342943e"} Jul 16 04:39:15.455710 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.455622 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 04:39:15.727853 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.726943 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:15.727853 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.727002 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:15.727853 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:15.727186 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 04:39:15.727853 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:15.727205 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 04:39:15.727853 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:15.727218 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w69nk for pod openshift-network-diagnostics/network-check-target-lgzrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:15.727853 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:15.727289 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk podName:255f6da7-bab4-4b7c-984f-cd1ddda313e9 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:17.727256883 +0000 UTC m=+5.045305439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w69nk" (UniqueName: "kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk") pod "network-check-target-lgzrj" (UID: "255f6da7-bab4-4b7c-984f-cd1ddda313e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:15.727853 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:15.727367 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:15.727853 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:15.727403 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs podName:42f5e13d-20f3-46ed-ab64-13ce54e5e719 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:17.727390677 +0000 UTC m=+5.045439228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs") pod "network-metrics-daemon-nsrxl" (UID: "42f5e13d-20f3-46ed-ab64-13ce54e5e719") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:15.790676 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.790467 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 04:39:15.827591 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:15.827511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:15.827781 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:15.827666 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:15.827781 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:15.827732 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret podName:2385ede2-8c97-424e-b560-19011a292b9e nodeName:}" failed. No retries permitted until 2026-07-16 04:39:17.827713293 +0000 UTC m=+5.145761831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret") pod "global-pull-secret-syncer-l6277" (UID: "2385ede2-8c97-424e-b560-19011a292b9e") : object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:16.157228 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:16.157110 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 04:34:14 +0000 UTC" deadline="2028-02-20 14:29:27.740170621 +0000 UTC" Jul 16 04:39:16.157228 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:16.157175 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14025h50m11.582999782s" Jul 16 04:39:16.204651 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:16.204487 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:16.204651 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:16.204612 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:16.205065 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:16.205036 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:16.205281 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:16.205133 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:17.203913 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:17.203886 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:17.204371 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:17.204024 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:17.743937 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:17.743902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:17.744113 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:17.743950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:17.744171 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:17.744112 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 04:39:17.744171 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:17.744130 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 04:39:17.744171 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:17.744142 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w69nk for pod openshift-network-diagnostics/network-check-target-lgzrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:17.744316 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:17.744194 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk podName:255f6da7-bab4-4b7c-984f-cd1ddda313e9 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:21.744177581 +0000 UTC m=+9.062226133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w69nk" (UniqueName: "kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk") pod "network-check-target-lgzrj" (UID: "255f6da7-bab4-4b7c-984f-cd1ddda313e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:17.744609 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:17.744590 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:17.744667 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:17.744647 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs podName:42f5e13d-20f3-46ed-ab64-13ce54e5e719 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:21.744630929 +0000 UTC m=+9.062679470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs") pod "network-metrics-daemon-nsrxl" (UID: "42f5e13d-20f3-46ed-ab64-13ce54e5e719") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:17.845858 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:17.845250 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:17.845858 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:17.845419 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:17.845858 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:17.845476 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret podName:2385ede2-8c97-424e-b560-19011a292b9e nodeName:}" failed. No retries permitted until 2026-07-16 04:39:21.845456831 +0000 UTC m=+9.163505381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret") pod "global-pull-secret-syncer-l6277" (UID: "2385ede2-8c97-424e-b560-19011a292b9e") : object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:18.204724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:18.204270 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:18.204724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:18.204303 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:18.204724 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:18.204390 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:18.204724 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:18.204498 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:19.204483 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:19.204381 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:19.204644 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:19.204526 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:20.203476 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:20.203443 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:20.203982 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:20.203574 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:20.204049 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:20.203984 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:20.204103 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:20.204081 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:21.204904 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:21.204853 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:21.205353 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:21.205039 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:21.782421 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:21.781684 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:21.782421 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:21.781746 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:21.782421 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:21.781939 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 04:39:21.782421 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:21.781972 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 04:39:21.782421 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:21.781986 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w69nk for pod openshift-network-diagnostics/network-check-target-lgzrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:21.782421 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:21.782000 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:21.782421 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:21.782052 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk podName:255f6da7-bab4-4b7c-984f-cd1ddda313e9 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:29.782032917 +0000 UTC m=+17.100081468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w69nk" (UniqueName: "kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk") pod "network-check-target-lgzrj" (UID: "255f6da7-bab4-4b7c-984f-cd1ddda313e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:21.782421 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:21.782074 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs podName:42f5e13d-20f3-46ed-ab64-13ce54e5e719 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:29.782063891 +0000 UTC m=+17.100112428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs") pod "network-metrics-daemon-nsrxl" (UID: "42f5e13d-20f3-46ed-ab64-13ce54e5e719") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:21.883230 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:21.883152 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:21.883383 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:21.883290 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:21.883383 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:21.883347 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret podName:2385ede2-8c97-424e-b560-19011a292b9e nodeName:}" failed. No retries permitted until 2026-07-16 04:39:29.883329978 +0000 UTC m=+17.201378520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret") pod "global-pull-secret-syncer-l6277" (UID: "2385ede2-8c97-424e-b560-19011a292b9e") : object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:22.203782 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:22.203686 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:22.203932 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:22.203833 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:22.203932 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:22.203686 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:22.203932 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:22.203922 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:23.205378 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:23.205344 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:23.205863 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:23.205477 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:24.203620 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:24.203569 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:24.203805 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:24.203587 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:24.203805 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:24.203703 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:24.203805 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:24.203784 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:25.203866 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:25.203836 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:25.204256 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:25.203963 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:26.203748 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:26.203719 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:26.203895 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:26.203727 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:26.203895 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:26.203853 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:26.204283 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:26.203917 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:27.204475 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:27.204447 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:27.204955 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:27.204578 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:28.204240 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:28.204201 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:28.204423 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:28.204202 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:28.204423 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:28.204330 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:28.204423 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:28.204384 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:29.204103 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:29.204071 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:29.204285 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:29.204207 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:29.845867 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:29.845841 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:29.846200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:29.845874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:29.846200 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:29.845987 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:29.846200 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:29.845996 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 04:39:29.846200 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:29.846010 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 04:39:29.846200 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:29.846020 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w69nk for pod openshift-network-diagnostics/network-check-target-lgzrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:29.846200 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:29.846046 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs podName:42f5e13d-20f3-46ed-ab64-13ce54e5e719 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:45.846029821 +0000 UTC m=+33.164078364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs") pod "network-metrics-daemon-nsrxl" (UID: "42f5e13d-20f3-46ed-ab64-13ce54e5e719") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:29.846200 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:29.846061 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk podName:255f6da7-bab4-4b7c-984f-cd1ddda313e9 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:45.846054662 +0000 UTC m=+33.164103199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w69nk" (UniqueName: "kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk") pod "network-check-target-lgzrj" (UID: "255f6da7-bab4-4b7c-984f-cd1ddda313e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:29.946980 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:29.946958 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:29.947091 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:29.947079 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:29.947134 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:29.947128 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret podName:2385ede2-8c97-424e-b560-19011a292b9e nodeName:}" failed. No retries permitted until 2026-07-16 04:39:45.947114772 +0000 UTC m=+33.265163309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret") pod "global-pull-secret-syncer-l6277" (UID: "2385ede2-8c97-424e-b560-19011a292b9e") : object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:30.204126 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:30.204092 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:30.204315 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:30.204092 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:30.204315 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:30.204227 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:30.204431 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:30.204315 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:31.203707 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:31.203667 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:31.204161 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:31.203830 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:32.204078 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:32.204036 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:32.204528 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:32.204181 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:32.204528 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:32.204250 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:32.204528 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:32.204376 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:33.204444 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:33.204412 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:33.204904 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:33.204523 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:34.203614 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:34.203580 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:34.203808 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:34.203710 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:34.203808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:34.203779 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:34.203899 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:34.203870 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:34.419046 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:34.418872 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" event={"ID":"b2d100063295820fbe31810c8234ecb8","Type":"ContainerStarted","Data":"27a97a44470ac9a050cd903eafe2625013be9c97aa3f697bc99c4d5ce4b3ab94"} Jul 16 04:39:34.419379 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:34.419073 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:34.437552 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:34.437530 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 04:39:34.438203 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:34.438183 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal"] Jul 16 04:39:35.203679 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.203540 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:35.203818 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:35.203799 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:35.421846 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.421635 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tqh7x" event={"ID":"0e63d7c4-a323-4925-a1e2-06690436efe2","Type":"ContainerStarted","Data":"99a4f18427b9e75327c0bb513549f189a7053679a9f5b5110e54d6bc161145cc"} Jul 16 04:39:35.422995 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.422976 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" event={"ID":"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3","Type":"ContainerStarted","Data":"b298778dda9ba9cb4d8a24318ae6bfb0e667534dbe5728a4de4e9ad02708988b"} Jul 16 04:39:35.424121 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.424102 2585 generic.go:356] "Generic (PLEG): container finished" podID="48c4ca63-e0a4-4cb0-be52-2476b3758d98" containerID="a8cf5087bb79c69e4c1cdccddfa7232aff14d0fc110b5386bc42bdad354a3019" exitCode=0 Jul 16 04:39:35.424211 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.424161 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55n8f" event={"ID":"48c4ca63-e0a4-4cb0-be52-2476b3758d98","Type":"ContainerDied","Data":"a8cf5087bb79c69e4c1cdccddfa7232aff14d0fc110b5386bc42bdad354a3019"} Jul 16 04:39:35.425402 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.425338 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4wmg9" event={"ID":"00a467c5-6e94-4450-b9f0-c84d4e1de57c","Type":"ContainerStarted","Data":"a18b16ba64dca2545247f5bb2dbbb77537a9878047e92781119c81515e3f91ef"} Jul 16 04:39:35.426578 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.426560 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nnfvz" event={"ID":"496d22a4-290a-4a92-9a34-b3d48db8a114","Type":"ContainerStarted","Data":"0ab52f53882c6eba4b405ff9a616e05e51d10b55332b88d971ec4d3e22ced257"} Jul 16 04:39:35.430453 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.430378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cgjjz" event={"ID":"69a01906-2a04-452a-b44e-9f6971572c5d","Type":"ContainerStarted","Data":"a85119226edea64251150569b289d1f7dd22c24f8bcc55401b9e713a6d994f8c"} Jul 16 04:39:35.432251 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.432229 2585 generic.go:356] "Generic (PLEG): container finished" podID="e834723c39bbc8029a7518c929607dfd" containerID="3cb9defba2b31c1eb90de2f2a46d922f2d099f6a7eb32137e741d9261d9869a0" exitCode=0 Jul 16 04:39:35.432337 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.432294 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" event={"ID":"e834723c39bbc8029a7518c929607dfd","Type":"ContainerDied","Data":"3cb9defba2b31c1eb90de2f2a46d922f2d099f6a7eb32137e741d9261d9869a0"} Jul 16 04:39:35.435471 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.435452 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:39:35.435861 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.435838 2585 generic.go:356] "Generic (PLEG): container finished" podID="1258efcc-2979-4c2a-a9a3-26f06d1699bd" containerID="11d2bd37cef105de2967c66005bd4d84082cae486933c329a92380e06e4ad6d1" exitCode=1 Jul 16 04:39:35.435946 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.435905 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerStarted","Data":"cc52f839ea18f1a1b4eb071afcc10171ecc0a4a2d59ca147d34a5f8942deba4b"} Jul 16 04:39:35.436008 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.435969 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerStarted","Data":"532d4d7a1dda3eaddcfae2a513b6078b613b8020cbdd40637beeb335f7fe2f36"} Jul 16 04:39:35.436059 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.436010 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerStarted","Data":"659088dfdc001e6aa9f4eef6d4b11d649a567e81c8e88ea41ebbf828680971c4"} Jul 16 04:39:35.436059 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.436051 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerStarted","Data":"cd54f239851f44c0d4e002224011659ec07a488fcbe3284d1208b5e03b289db8"} Jul 16 04:39:35.436131 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.436083 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerDied","Data":"11d2bd37cef105de2967c66005bd4d84082cae486933c329a92380e06e4ad6d1"} Jul 16 04:39:35.436166 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.436129 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerStarted","Data":"adcfe510b4a62c3eebc3d6d5bd2abb4ee059633dce3cc3940415a2e65f67cf05"} Jul 16 04:39:35.437201 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.437183 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-szbxb" event={"ID":"3680ab79-d032-4fac-8abe-f125045074cd","Type":"ContainerStarted","Data":"68581716204a95aa648290f0ec11163c1e7b90f0137cd37976521251834785ef"} Jul 16 04:39:35.437360 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.437345 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:35.438143 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.438093 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" podStartSLOduration=1.438077474 podStartE2EDuration="1.438077474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:35.437875191 +0000 UTC m=+22.755923749" watchObservedRunningTime="2026-07-16 04:39:35.438077474 +0000 UTC m=+22.756126056" Jul 16 04:39:35.449237 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.449220 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 04:39:35.449310 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:35.449256 2585 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-3.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-3.ec2.internal" Jul 16 04:39:35.455899 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.455863 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tqh7x" podStartSLOduration=2.831471933 podStartE2EDuration="22.45585156s" totalImagesPullingTime="19.624379627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:35.455553343 +0000 UTC m=+22.773601903" watchObservedRunningTime="2026-07-16 04:39:35.45585156 +0000 UTC m=+22.773900117" Jul 16 04:39:35.474742 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.474646 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-szbxb" podStartSLOduration=2.602167755 podStartE2EDuration="22.474631838s" totalImagesPullingTime="19.872464083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:35.474221663 +0000 UTC m=+22.792270221" watchObservedRunningTime="2026-07-16 04:39:35.474631838 +0000 UTC m=+22.792680396" Jul 16 04:39:35.517288 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.517251 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nnfvz" podStartSLOduration=2.685475287 podStartE2EDuration="22.517236324s" totalImagesPullingTime="19.831761037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:35.516351751 +0000 UTC m=+22.834400310" watchObservedRunningTime="2026-07-16 04:39:35.517236324 +0000 UTC m=+22.835284881" Jul 16 04:39:35.536591 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.536553 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cgjjz" podStartSLOduration=2.812379929 podStartE2EDuration="22.536538754s" totalImagesPullingTime="19.724158825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:35.536283835 +0000 UTC m=+22.854332393" watchObservedRunningTime="2026-07-16 04:39:35.536538754 +0000 UTC m=+22.854587322" Jul 16 04:39:35.559249 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:35.559207 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4wmg9" podStartSLOduration=2.768426041 podStartE2EDuration="22.559192314s" totalImagesPullingTime="19.790766273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:35.559044424 +0000 UTC m=+22.877093131" watchObservedRunningTime="2026-07-16 04:39:35.559192314 +0000 UTC m=+22.877240872" Jul 16 04:39:36.204147 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:36.204118 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:36.204327 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:36.204157 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:36.204327 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:36.204256 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:36.204432 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:36.204384 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:36.441343 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:36.441303 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" event={"ID":"e834723c39bbc8029a7518c929607dfd","Type":"ContainerStarted","Data":"fd359bc2421fd42984fc4b0f1e0888b2d9e636feee7dcc7eedf9f462f2922dc6"} Jul 16 04:39:36.442725 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:36.442699 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7887n" event={"ID":"3dd29329-28ee-4067-beda-dc5dc41c707a","Type":"ContainerStarted","Data":"457df5ec518b35d9e6561edf3c8b630b686e1887d6c6fe1a31d2062e9d3c6f4d"} Jul 16 04:39:36.558730 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:36.558630 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-3.ec2.internal" podStartSLOduration=22.558610452 podStartE2EDuration="22.558610452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:36.467669864 +0000 UTC m=+23.785718442" watchObservedRunningTime="2026-07-16 04:39:36.558610452 +0000 UTC m=+23.876659010" Jul 16 04:39:36.717312 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:36.717283 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 04:39:37.204510 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:37.204476 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:37.204800 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:37.204620 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:37.216670 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:37.216564 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T04:39:36.7173072Z","UUID":"af3c4c46-1cda-4072-a9fa-a86192547c3d","Handler":null,"Name":"","Endpoint":""} Jul 16 04:39:37.218468 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:37.218445 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 04:39:37.218596 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:37.218486 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 04:39:37.447651 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:37.447598 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" event={"ID":"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3","Type":"ContainerStarted","Data":"5c3fbcb395ca5527eeafda086cbb6a862451ae61bd6cadecb6157187008e9c48"} Jul 16 04:39:37.641383 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:37.641359 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:37.641997 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:37.641978 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:37.664534 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:37.664494 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7887n" podStartSLOduration=5.035015948 podStartE2EDuration="24.664478548s" totalImagesPullingTime="19.6294626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:36.559413699 +0000 UTC m=+23.877462257" watchObservedRunningTime="2026-07-16 04:39:37.664478548 +0000 UTC m=+24.982527107" Jul 16 04:39:38.203942 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:38.203873 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:38.203942 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:38.203905 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:38.204211 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:38.204013 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:38.204211 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:38.204119 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:38.453487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:38.453453 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:39:38.454010 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:38.453894 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerStarted","Data":"d073aa4774691c0ff07aa43a03098ef891c415d3b0a85ada1aac370e624897ae"} Jul 16 04:39:38.456226 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:38.456199 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" event={"ID":"0a8966ce-dc73-4a6d-94cb-f2430c7dddd3","Type":"ContainerStarted","Data":"1758d48877a28c054d0300a4df2461adf71a8bb40d109525359d7e9cc074c4e0"} Jul 16 04:39:38.456449 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:38.456429 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:38.456958 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:38.456941 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nnfvz" Jul 16 04:39:38.517340 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:38.517286 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prw5c" podStartSLOduration=3.107592563 podStartE2EDuration="25.51726789s" totalImagesPullingTime="22.409675327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:38.495040874 +0000 UTC m=+25.813089453" watchObservedRunningTime="2026-07-16 04:39:38.51726789 +0000 UTC m=+25.835316447" Jul 16 04:39:39.204115 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:39.203910 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:39.204309 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:39.204253 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:40.203954 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.203882 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:40.204574 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:40.203973 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:40.204574 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.204026 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:40.204574 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:40.204140 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:40.463126 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.462921 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:39:40.463429 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.463405 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerStarted","Data":"cd154015e5415f5fbf13fb6afaf5384da23d1bb1f628d779f22e9753b08be8e8"} Jul 16 04:39:40.463664 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.463638 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:40.463818 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.463694 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:40.463893 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.463819 2585 scope.go:117] "RemoveContainer" containerID="11d2bd37cef105de2967c66005bd4d84082cae486933c329a92380e06e4ad6d1" Jul 16 04:39:40.465290 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.465271 2585 generic.go:356] "Generic (PLEG): container finished" podID="48c4ca63-e0a4-4cb0-be52-2476b3758d98" containerID="6a881b81177f0a1368607db5a328698d33d6af627f16790cd184e3377997dda4" exitCode=0 Jul 16 04:39:40.465372 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.465351 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55n8f" event={"ID":"48c4ca63-e0a4-4cb0-be52-2476b3758d98","Type":"ContainerDied","Data":"6a881b81177f0a1368607db5a328698d33d6af627f16790cd184e3377997dda4"} Jul 16 04:39:40.479862 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:40.479760 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:41.204270 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:41.204240 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:41.204681 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:41.204350 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:41.470705 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:41.470643 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:39:41.470946 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:41.470923 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" event={"ID":"1258efcc-2979-4c2a-a9a3-26f06d1699bd","Type":"ContainerStarted","Data":"f628922b0f07f014ee200f54d4b810e7e47f155e708795fb646c3212c5b3d3b2"} Jul 16 04:39:41.471105 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:41.471090 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:41.485418 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:41.485393 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:39:41.553634 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:41.553588 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" podStartSLOduration=8.321393614 podStartE2EDuration="28.553569834s" totalImagesPullingTime="20.23217622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:41.510961244 +0000 UTC m=+28.829009821" watchObservedRunningTime="2026-07-16 04:39:41.553569834 +0000 UTC m=+28.871618429" Jul 16 04:39:42.204043 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.204011 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:42.204212 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.204029 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:42.204212 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:42.204113 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:42.204341 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:42.204205 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:42.237230 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.237204 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6277"] Jul 16 04:39:42.250761 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.250703 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lgzrj"] Jul 16 04:39:42.273038 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.273005 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nsrxl"] Jul 16 04:39:42.273169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.273098 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:42.273226 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:42.273191 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:42.475361 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.475330 2585 generic.go:356] "Generic (PLEG): container finished" podID="48c4ca63-e0a4-4cb0-be52-2476b3758d98" containerID="6ec2c76611c7aef1b78a82f11059a921c5451b85d41f52a006c422f173b66428" exitCode=0 Jul 16 04:39:42.475515 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.475410 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:42.475515 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.475432 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55n8f" event={"ID":"48c4ca63-e0a4-4cb0-be52-2476b3758d98","Type":"ContainerDied","Data":"6ec2c76611c7aef1b78a82f11059a921c5451b85d41f52a006c422f173b66428"} Jul 16 04:39:42.475515 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:42.475478 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:42.475644 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:42.475558 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:42.475748 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:42.475691 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:43.480274 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:43.480095 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55n8f" event={"ID":"48c4ca63-e0a4-4cb0-be52-2476b3758d98","Type":"ContainerStarted","Data":"214d3097d303438f8e4dc218e6abaa1b8444ae0827f8b8f10eb2b4e1edc7cd26"} Jul 16 04:39:44.204467 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:44.204432 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:44.204603 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:44.204447 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:44.204669 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:44.204453 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:44.204669 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:44.204587 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:44.204669 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:44.204650 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:44.204846 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:44.204754 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:44.484687 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:44.484607 2585 generic.go:356] "Generic (PLEG): container finished" podID="48c4ca63-e0a4-4cb0-be52-2476b3758d98" containerID="214d3097d303438f8e4dc218e6abaa1b8444ae0827f8b8f10eb2b4e1edc7cd26" exitCode=0 Jul 16 04:39:44.484687 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:44.484662 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55n8f" event={"ID":"48c4ca63-e0a4-4cb0-be52-2476b3758d98","Type":"ContainerDied","Data":"214d3097d303438f8e4dc218e6abaa1b8444ae0827f8b8f10eb2b4e1edc7cd26"} Jul 16 04:39:45.869388 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:45.869349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:45.869923 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:45.869393 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:45.869923 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:45.869503 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:45.869923 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:45.869513 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 04:39:45.869923 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:45.869527 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 04:39:45.869923 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:45.869538 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w69nk for pod openshift-network-diagnostics/network-check-target-lgzrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:45.869923 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:45.869572 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs podName:42f5e13d-20f3-46ed-ab64-13ce54e5e719 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:17.869553471 +0000 UTC m=+65.187602028 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs") pod "network-metrics-daemon-nsrxl" (UID: "42f5e13d-20f3-46ed-ab64-13ce54e5e719") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 04:39:45.869923 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:45.869588 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk podName:255f6da7-bab4-4b7c-984f-cd1ddda313e9 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:17.869580715 +0000 UTC m=+65.187629253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w69nk" (UniqueName: "kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk") pod "network-check-target-lgzrj" (UID: "255f6da7-bab4-4b7c-984f-cd1ddda313e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 04:39:45.970221 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:45.970184 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:45.970383 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:45.970350 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:45.970440 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:45.970427 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret podName:2385ede2-8c97-424e-b560-19011a292b9e nodeName:}" failed. No retries permitted until 2026-07-16 04:40:17.970406803 +0000 UTC m=+65.288455347 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret") pod "global-pull-secret-syncer-l6277" (UID: "2385ede2-8c97-424e-b560-19011a292b9e") : object "kube-system"/"original-pull-secret" not registered Jul 16 04:39:46.204444 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:46.204358 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:46.204444 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:46.204407 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:46.204669 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:46.204472 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:46.204669 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:46.204488 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6277" podUID="2385ede2-8c97-424e-b560-19011a292b9e" Jul 16 04:39:46.204669 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:46.204552 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lgzrj" podUID="255f6da7-bab4-4b7c-984f-cd1ddda313e9" Jul 16 04:39:46.204669 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:46.204645 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrxl" podUID="42f5e13d-20f3-46ed-ab64-13ce54e5e719" Jul 16 04:39:47.515716 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.515690 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-3.ec2.internal" event="NodeReady" Jul 16 04:39:47.516254 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.515840 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 04:39:47.569332 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.569303 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-757q2"] Jul 16 04:39:47.573818 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.573795 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.577370 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.577349 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 04:39:47.577477 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.577413 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pmfl6\"" Jul 16 04:39:47.577956 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.577935 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 04:39:47.578388 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.578365 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qpks8"] Jul 16 04:39:47.581745 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.581635 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qpks8" Jul 16 04:39:47.586200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.586180 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 04:39:47.586411 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.586391 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 04:39:47.586532 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.586498 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 04:39:47.592351 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.592332 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ggd6f\"" Jul 16 04:39:47.593152 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.593121 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-757q2"] Jul 16 04:39:47.596328 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.596309 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qpks8"] Jul 16 04:39:47.681262 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.681232 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c56c68-9f4a-445b-9739-c33a31e7def0-tmp-dir\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.681445 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.681293 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66-cert\") pod \"ingress-canary-qpks8\" (UID: \"f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66\") " pod="openshift-ingress-canary/ingress-canary-qpks8" Jul 16 04:39:47.681445 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.681320 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkdq\" (UniqueName: \"kubernetes.io/projected/f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66-kube-api-access-8mkdq\") pod \"ingress-canary-qpks8\" (UID: \"f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66\") " pod="openshift-ingress-canary/ingress-canary-qpks8" Jul 16 04:39:47.681445 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.681411 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.681575 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.681446 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28d7v\" (UniqueName: \"kubernetes.io/projected/a1c56c68-9f4a-445b-9739-c33a31e7def0-kube-api-access-28d7v\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.681575 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.681491 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c56c68-9f4a-445b-9739-c33a31e7def0-config-volume\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.782140 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.782057 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c56c68-9f4a-445b-9739-c33a31e7def0-config-volume\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.782140 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.782098 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c56c68-9f4a-445b-9739-c33a31e7def0-tmp-dir\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.782350 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.782242 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66-cert\") pod \"ingress-canary-qpks8\" (UID: \"f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66\") " pod="openshift-ingress-canary/ingress-canary-qpks8" Jul 16 04:39:47.782350 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.782285 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkdq\" (UniqueName: \"kubernetes.io/projected/f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66-kube-api-access-8mkdq\") pod \"ingress-canary-qpks8\" (UID: \"f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66\") " pod="openshift-ingress-canary/ingress-canary-qpks8" Jul 16 04:39:47.782455 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.782353 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.782455 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.782382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28d7v\" (UniqueName: \"kubernetes.io/projected/a1c56c68-9f4a-445b-9739-c33a31e7def0-kube-api-access-28d7v\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.782546 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.782485 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c56c68-9f4a-445b-9739-c33a31e7def0-tmp-dir\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.782944 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.782650 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c56c68-9f4a-445b-9739-c33a31e7def0-config-volume\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.782944 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:47.782669 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 04:39:47.782944 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:47.782736 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls podName:a1c56c68-9f4a-445b-9739-c33a31e7def0 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:48.282722843 +0000 UTC m=+35.600771416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls") pod "dns-default-757q2" (UID: "a1c56c68-9f4a-445b-9739-c33a31e7def0") : secret "dns-default-metrics-tls" not found Jul 16 04:39:47.787043 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.787019 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66-cert\") pod \"ingress-canary-qpks8\" (UID: \"f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66\") " pod="openshift-ingress-canary/ingress-canary-qpks8" Jul 16 04:39:47.802713 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.802690 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28d7v\" (UniqueName: \"kubernetes.io/projected/a1c56c68-9f4a-445b-9739-c33a31e7def0-kube-api-access-28d7v\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:47.802860 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.802845 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkdq\" (UniqueName: \"kubernetes.io/projected/f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66-kube-api-access-8mkdq\") pod \"ingress-canary-qpks8\" (UID: \"f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66\") " pod="openshift-ingress-canary/ingress-canary-qpks8" Jul 16 04:39:47.894033 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:47.893997 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qpks8" Jul 16 04:39:48.203710 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.203670 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:39:48.203904 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.203689 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:39:48.203904 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.203814 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:39:48.210832 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.210810 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 04:39:48.211097 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.211075 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 04:39:48.211212 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.211075 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 04:39:48.211212 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.211082 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rjh6n\"" Jul 16 04:39:48.211425 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.211409 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 04:39:48.215897 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.215877 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6n555\"" Jul 16 04:39:48.286508 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.286480 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:48.286684 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:48.286598 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 04:39:48.286684 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:48.286676 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls podName:a1c56c68-9f4a-445b-9739-c33a31e7def0 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:49.286658969 +0000 UTC m=+36.604707521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls") pod "dns-default-757q2" (UID: "a1c56c68-9f4a-445b-9739-c33a31e7def0") : secret "dns-default-metrics-tls" not found Jul 16 04:39:48.695092 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:48.694937 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qpks8"] Jul 16 04:39:48.697497 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:39:48.697476 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bf91ab_7bc9_4ccf_b5ef_bfbfbfa6bf66.slice/crio-591e181020c9d1c2286ba0d09b769a8b0938173f461fbf13a8f4e5c80f05754b WatchSource:0}: Error finding container 591e181020c9d1c2286ba0d09b769a8b0938173f461fbf13a8f4e5c80f05754b: Status 404 returned error can't find the container with id 591e181020c9d1c2286ba0d09b769a8b0938173f461fbf13a8f4e5c80f05754b Jul 16 04:39:49.294041 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:49.294012 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:49.294207 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:49.294127 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 04:39:49.294207 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:49.294182 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls podName:a1c56c68-9f4a-445b-9739-c33a31e7def0 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:51.294167408 +0000 UTC m=+38.612215944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls") pod "dns-default-757q2" (UID: "a1c56c68-9f4a-445b-9739-c33a31e7def0") : secret "dns-default-metrics-tls" not found Jul 16 04:39:49.498417 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:49.498365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qpks8" event={"ID":"f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66","Type":"ContainerStarted","Data":"591e181020c9d1c2286ba0d09b769a8b0938173f461fbf13a8f4e5c80f05754b"} Jul 16 04:39:49.501306 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:49.501281 2585 generic.go:356] "Generic (PLEG): container finished" podID="48c4ca63-e0a4-4cb0-be52-2476b3758d98" containerID="e70270ae34747ca908e9e00d9705385c6bd398b69b41781b7d70f6fc7ab726bf" exitCode=0 Jul 16 04:39:49.501431 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:49.501327 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55n8f" event={"ID":"48c4ca63-e0a4-4cb0-be52-2476b3758d98","Type":"ContainerDied","Data":"e70270ae34747ca908e9e00d9705385c6bd398b69b41781b7d70f6fc7ab726bf"} Jul 16 04:39:50.508421 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:50.508367 2585 generic.go:356] "Generic (PLEG): container finished" podID="48c4ca63-e0a4-4cb0-be52-2476b3758d98" containerID="8adbcff7e5d1829ff1027e07a7916266bdfcff5c6039227f8ec0b7d67a658a9b" exitCode=0 Jul 16 04:39:50.508862 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:50.508427 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55n8f" event={"ID":"48c4ca63-e0a4-4cb0-be52-2476b3758d98","Type":"ContainerDied","Data":"8adbcff7e5d1829ff1027e07a7916266bdfcff5c6039227f8ec0b7d67a658a9b"} Jul 16 04:39:51.309980 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:51.309756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:51.310124 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:51.309913 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 04:39:51.310124 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:51.310094 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls podName:a1c56c68-9f4a-445b-9739-c33a31e7def0 nodeName:}" failed. No retries permitted until 2026-07-16 04:39:55.310078917 +0000 UTC m=+42.628127453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls") pod "dns-default-757q2" (UID: "a1c56c68-9f4a-445b-9739-c33a31e7def0") : secret "dns-default-metrics-tls" not found Jul 16 04:39:51.511714 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:51.511629 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qpks8" event={"ID":"f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66","Type":"ContainerStarted","Data":"254629e4c000a63151f637b398cfcd48356b20be4b74ebcd22e0dd580beba84d"} Jul 16 04:39:51.514631 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:51.514597 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55n8f" event={"ID":"48c4ca63-e0a4-4cb0-be52-2476b3758d98","Type":"ContainerStarted","Data":"4cb0b06f0ce960da7b8151efdaa1c50703aee15e900bd886126897c964b30ce6"} Jul 16 04:39:51.532172 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:51.532131 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qpks8" podStartSLOduration=2.147810782 podStartE2EDuration="4.532118073s" totalImagesPullingTime="2.384307291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:51.530861583 +0000 UTC m=+38.848910152" watchObservedRunningTime="2026-07-16 04:39:51.532118073 +0000 UTC m=+38.850166612" Jul 16 04:39:51.558223 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:51.558173 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-55n8f" podStartSLOduration=8.70998396 podStartE2EDuration="38.558159594s" totalImagesPullingTime="29.848175634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:39:51.55772537 +0000 UTC m=+38.875773929" watchObservedRunningTime="2026-07-16 04:39:51.558159594 +0000 UTC m=+38.876208153" Jul 16 04:39:55.336147 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:39:55.336107 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:39:55.336611 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:55.336217 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 04:39:55.336611 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:39:55.336277 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls podName:a1c56c68-9f4a-445b-9739-c33a31e7def0 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:03.336262205 +0000 UTC m=+50.654310741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls") pod "dns-default-757q2" (UID: "a1c56c68-9f4a-445b-9739-c33a31e7def0") : secret "dns-default-metrics-tls" not found Jul 16 04:40:03.390476 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:03.390429 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:40:03.390965 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:03.390563 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 04:40:03.390965 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:03.390612 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls podName:a1c56c68-9f4a-445b-9739-c33a31e7def0 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:19.390598932 +0000 UTC m=+66.708647468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls") pod "dns-default-757q2" (UID: "a1c56c68-9f4a-445b-9739-c33a31e7def0") : secret "dns-default-metrics-tls" not found Jul 16 04:40:11.780070 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.780027 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c"] Jul 16 04:40:11.781859 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.781844 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:11.790005 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.789983 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-mlq84"] Jul 16 04:40:11.791575 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.791561 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.795201 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.795183 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 04:40:11.797516 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.797503 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 04:40:11.797587 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.797519 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-w2tj7\"" Jul 16 04:40:11.797587 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.797519 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 16 04:40:11.797662 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.797501 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 16 04:40:11.811938 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.811915 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 16 04:40:11.812246 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.812230 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 16 04:40:11.812663 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.812618 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 16 04:40:11.812779 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.812236 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 16 04:40:11.815598 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.815574 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-27l5b\"" Jul 16 04:40:11.824843 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.824811 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mlq84"] Jul 16 04:40:11.832031 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.832010 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c"] Jul 16 04:40:11.835441 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.835420 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 16 04:40:11.847471 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.847447 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/be0c3d73-85fb-40a9-a2af-001e8231724a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:11.847550 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.847484 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:11.847550 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.847503 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9870a155-1b41-4300-9007-f16f24f9c3cd-serving-cert\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.847550 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.847518 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqzz2\" (UniqueName: \"kubernetes.io/projected/9870a155-1b41-4300-9007-f16f24f9c3cd-kube-api-access-nqzz2\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.847673 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.847563 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9870a155-1b41-4300-9007-f16f24f9c3cd-config\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.847673 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.847610 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkq8m\" (UniqueName: \"kubernetes.io/projected/be0c3d73-85fb-40a9-a2af-001e8231724a-kube-api-access-nkq8m\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:11.847673 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.847652 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9870a155-1b41-4300-9007-f16f24f9c3cd-trusted-ca\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.929876 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.929831 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7b679c4ddd-d64dk"] Jul 16 04:40:11.932834 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.932819 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:11.936647 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.936631 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hprbf\"" Jul 16 04:40:11.937470 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.937448 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 16 04:40:11.937571 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.937449 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 16 04:40:11.937805 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.937791 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 16 04:40:11.937956 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.937939 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 04:40:11.938128 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.938087 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 04:40:11.948526 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948505 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-stats-auth\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:11.948639 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948563 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/be0c3d73-85fb-40a9-a2af-001e8231724a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:11.948639 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948570 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 16 04:40:11.948639 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948580 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:11.948639 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948596 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:11.948639 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:11.948933 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948673 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9870a155-1b41-4300-9007-f16f24f9c3cd-serving-cert\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.948933 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:11.948700 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:11.948933 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948704 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqzz2\" (UniqueName: \"kubernetes.io/projected/9870a155-1b41-4300-9007-f16f24f9c3cd-kube-api-access-nqzz2\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.948933 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-default-certificate\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:11.948933 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:11.948756 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls podName:be0c3d73-85fb-40a9-a2af-001e8231724a nodeName:}" failed. No retries permitted until 2026-07-16 04:40:12.448737825 +0000 UTC m=+59.766786361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-g544c" (UID: "be0c3d73-85fb-40a9-a2af-001e8231724a") : secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:11.948933 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948833 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9870a155-1b41-4300-9007-f16f24f9c3cd-config\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.948933 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948858 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbwmf\" (UniqueName: \"kubernetes.io/projected/fcf7a78a-b00a-4419-923e-5cbde11f9f85-kube-api-access-gbwmf\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:11.948933 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948914 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkq8m\" (UniqueName: \"kubernetes.io/projected/be0c3d73-85fb-40a9-a2af-001e8231724a-kube-api-access-nkq8m\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:11.948933 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.948938 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9870a155-1b41-4300-9007-f16f24f9c3cd-trusted-ca\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.949344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.949284 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/be0c3d73-85fb-40a9-a2af-001e8231724a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:11.949486 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.949470 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9870a155-1b41-4300-9007-f16f24f9c3cd-config\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.949563 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.949549 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9870a155-1b41-4300-9007-f16f24f9c3cd-trusted-ca\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.952711 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.952694 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9870a155-1b41-4300-9007-f16f24f9c3cd-serving-cert\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:11.981948 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.981925 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b679c4ddd-d64dk"] Jul 16 04:40:11.987122 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:11.987101 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkq8m\" (UniqueName: \"kubernetes.io/projected/be0c3d73-85fb-40a9-a2af-001e8231724a-kube-api-access-nkq8m\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:12.007846 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.007822 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqzz2\" (UniqueName: \"kubernetes.io/projected/9870a155-1b41-4300-9007-f16f24f9c3cd-kube-api-access-nqzz2\") pod \"console-operator-7866497494-mlq84\" (UID: \"9870a155-1b41-4300-9007-f16f24f9c3cd\") " pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:12.050009 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.049984 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.050009 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.050014 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.050202 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.050055 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-default-certificate\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.050202 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.050090 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbwmf\" (UniqueName: \"kubernetes.io/projected/fcf7a78a-b00a-4419-923e-5cbde11f9f85-kube-api-access-gbwmf\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.050202 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.050136 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 04:40:12.050202 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.050169 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:12.550150635 +0000 UTC m=+59.868199193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : configmap references non-existent config key: service-ca.crt Jul 16 04:40:12.050202 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.050193 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:12.550180289 +0000 UTC m=+59.868228825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : secret "router-metrics-certs-default" not found Jul 16 04:40:12.050390 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.050209 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-stats-auth\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.052491 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.052467 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-stats-auth\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.052592 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.052514 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-default-certificate\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.080298 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.080269 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4"] Jul 16 04:40:12.081406 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.081386 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbwmf\" (UniqueName: \"kubernetes.io/projected/fcf7a78a-b00a-4419-923e-5cbde11f9f85-kube-api-access-gbwmf\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.083878 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.083861 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4" Jul 16 04:40:12.088317 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.088301 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-42gsj\"" Jul 16 04:40:12.098934 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.098910 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6"] Jul 16 04:40:12.099054 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.098979 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:12.118533 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.118510 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4"] Jul 16 04:40:12.118629 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.118608 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:12.125890 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.125872 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 16 04:40:12.126105 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.125873 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 16 04:40:12.126177 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.125916 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 16 04:40:12.126234 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.125951 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v5r5s\"" Jul 16 04:40:12.151054 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.151029 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:12.151182 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.151079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qzcp\" (UniqueName: \"kubernetes.io/projected/0798deea-6d3c-4db5-94b7-111ac1a48669-kube-api-access-4qzcp\") pod \"network-check-source-869dcfcdcd-g9vq4\" (UID: \"0798deea-6d3c-4db5-94b7-111ac1a48669\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4" Jul 16 04:40:12.151182 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.151149 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkcw5\" (UniqueName: \"kubernetes.io/projected/81dcab13-9c75-445a-a53a-7d6f05491ffd-kube-api-access-kkcw5\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:12.177814 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.177535 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6"] Jul 16 04:40:12.251676 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.251642 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qzcp\" (UniqueName: \"kubernetes.io/projected/0798deea-6d3c-4db5-94b7-111ac1a48669-kube-api-access-4qzcp\") pod \"network-check-source-869dcfcdcd-g9vq4\" (UID: \"0798deea-6d3c-4db5-94b7-111ac1a48669\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4" Jul 16 04:40:12.251862 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.251793 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkcw5\" (UniqueName: \"kubernetes.io/projected/81dcab13-9c75-445a-a53a-7d6f05491ffd-kube-api-access-kkcw5\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:12.251906 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.251866 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:12.251971 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.251957 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 04:40:12.252017 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.252010 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls podName:81dcab13-9c75-445a-a53a-7d6f05491ffd nodeName:}" failed. No retries permitted until 2026-07-16 04:40:12.75199161 +0000 UTC m=+60.070040147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls") pod "cluster-samples-operator-65db858894-c8kw6" (UID: "81dcab13-9c75-445a-a53a-7d6f05491ffd") : secret "samples-operator-tls" not found Jul 16 04:40:12.284173 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.284137 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mlq84"] Jul 16 04:40:12.287847 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:12.287821 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9870a155_1b41_4300_9007_f16f24f9c3cd.slice/crio-f22d532fa251474a7207544ac46100b48b0c69bcc19b4085dffaed0617057e9c WatchSource:0}: Error finding container f22d532fa251474a7207544ac46100b48b0c69bcc19b4085dffaed0617057e9c: Status 404 returned error can't find the container with id f22d532fa251474a7207544ac46100b48b0c69bcc19b4085dffaed0617057e9c Jul 16 04:40:12.289968 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.289948 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkcw5\" (UniqueName: \"kubernetes.io/projected/81dcab13-9c75-445a-a53a-7d6f05491ffd-kube-api-access-kkcw5\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:12.290031 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.289989 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qzcp\" (UniqueName: \"kubernetes.io/projected/0798deea-6d3c-4db5-94b7-111ac1a48669-kube-api-access-4qzcp\") pod \"network-check-source-869dcfcdcd-g9vq4\" (UID: \"0798deea-6d3c-4db5-94b7-111ac1a48669\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4" Jul 16 04:40:12.391796 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.391715 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4" Jul 16 04:40:12.453731 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.453697 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:12.453889 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.453868 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:12.453951 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.453940 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls podName:be0c3d73-85fb-40a9-a2af-001e8231724a nodeName:}" failed. No retries permitted until 2026-07-16 04:40:13.453922108 +0000 UTC m=+60.771970644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-g544c" (UID: "be0c3d73-85fb-40a9-a2af-001e8231724a") : secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:12.554515 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.554482 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.554515 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.554514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:12.554700 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.554617 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 04:40:12.554700 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.554666 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:13.554649762 +0000 UTC m=+60.872698299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : secret "router-metrics-certs-default" not found Jul 16 04:40:12.554795 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.554717 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:13.554708229 +0000 UTC m=+60.872756765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : configmap references non-existent config key: service-ca.crt Jul 16 04:40:12.558209 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.558184 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4"] Jul 16 04:40:12.562243 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:12.562215 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0798deea_6d3c_4db5_94b7_111ac1a48669.slice/crio-02f0e103aaa8dcc3a04221f840e275346db16f7eba0e6a270f0fe47d48b2fc0a WatchSource:0}: Error finding container 02f0e103aaa8dcc3a04221f840e275346db16f7eba0e6a270f0fe47d48b2fc0a: Status 404 returned error can't find the container with id 02f0e103aaa8dcc3a04221f840e275346db16f7eba0e6a270f0fe47d48b2fc0a Jul 16 04:40:12.562352 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.562313 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mlq84" event={"ID":"9870a155-1b41-4300-9007-f16f24f9c3cd","Type":"ContainerStarted","Data":"f22d532fa251474a7207544ac46100b48b0c69bcc19b4085dffaed0617057e9c"} Jul 16 04:40:12.755895 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:12.755819 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:12.756031 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.755965 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 04:40:12.756031 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:12.756031 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls podName:81dcab13-9c75-445a-a53a-7d6f05491ffd nodeName:}" failed. No retries permitted until 2026-07-16 04:40:13.75601134 +0000 UTC m=+61.074059884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls") pod "cluster-samples-operator-65db858894-c8kw6" (UID: "81dcab13-9c75-445a-a53a-7d6f05491ffd") : secret "samples-operator-tls" not found Jul 16 04:40:13.461557 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:13.461517 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:13.461976 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:13.461654 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:13.461976 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:13.461722 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls podName:be0c3d73-85fb-40a9-a2af-001e8231724a nodeName:}" failed. No retries permitted until 2026-07-16 04:40:15.461701818 +0000 UTC m=+62.779750357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-g544c" (UID: "be0c3d73-85fb-40a9-a2af-001e8231724a") : secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:13.502099 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:13.502070 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s27vv" Jul 16 04:40:13.562177 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:13.562143 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:13.562177 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:13.562183 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:13.562407 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:13.562320 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 04:40:13.562407 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:13.562389 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:15.56237048 +0000 UTC m=+62.880419033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : secret "router-metrics-certs-default" not found Jul 16 04:40:13.562689 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:13.562560 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:15.562547094 +0000 UTC m=+62.880595633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : configmap references non-existent config key: service-ca.crt Jul 16 04:40:13.565918 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:13.565895 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4" event={"ID":"0798deea-6d3c-4db5-94b7-111ac1a48669","Type":"ContainerStarted","Data":"02f0e103aaa8dcc3a04221f840e275346db16f7eba0e6a270f0fe47d48b2fc0a"} Jul 16 04:40:13.763736 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:13.763627 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:13.763907 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:13.763846 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 04:40:13.763974 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:13.763962 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls podName:81dcab13-9c75-445a-a53a-7d6f05491ffd nodeName:}" failed. No retries permitted until 2026-07-16 04:40:15.763936349 +0000 UTC m=+63.081984889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls") pod "cluster-samples-operator-65db858894-c8kw6" (UID: "81dcab13-9c75-445a-a53a-7d6f05491ffd") : secret "samples-operator-tls" not found Jul 16 04:40:14.570453 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:14.570424 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/0.log" Jul 16 04:40:14.570902 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:14.570469 2585 generic.go:356] "Generic (PLEG): container finished" podID="9870a155-1b41-4300-9007-f16f24f9c3cd" containerID="d337107b865a1bf9abce56f8269c87f856516deec9ddb3f78c6719f48bfc92d1" exitCode=255 Jul 16 04:40:14.570902 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:14.570500 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mlq84" event={"ID":"9870a155-1b41-4300-9007-f16f24f9c3cd","Type":"ContainerDied","Data":"d337107b865a1bf9abce56f8269c87f856516deec9ddb3f78c6719f48bfc92d1"} Jul 16 04:40:14.570902 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:14.570740 2585 scope.go:117] "RemoveContainer" containerID="d337107b865a1bf9abce56f8269c87f856516deec9ddb3f78c6719f48bfc92d1" Jul 16 04:40:15.476978 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.476942 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:15.477134 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:15.477109 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:15.477199 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:15.477188 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls podName:be0c3d73-85fb-40a9-a2af-001e8231724a nodeName:}" failed. No retries permitted until 2026-07-16 04:40:19.477165057 +0000 UTC m=+66.795213594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-g544c" (UID: "be0c3d73-85fb-40a9-a2af-001e8231724a") : secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:15.574825 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.574788 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4" event={"ID":"0798deea-6d3c-4db5-94b7-111ac1a48669","Type":"ContainerStarted","Data":"b2a90153b7cc39feed5e46624bde4d685372cc7f602e424506de8bf3a66279c7"} Jul 16 04:40:15.576317 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.576299 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:40:15.576721 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.576699 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/0.log" Jul 16 04:40:15.576872 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.576740 2585 generic.go:356] "Generic (PLEG): container finished" podID="9870a155-1b41-4300-9007-f16f24f9c3cd" containerID="29111863adc7c0d404a2d362b4cb169f4459a1cc0c7fb969be18fd067b7a3ec1" exitCode=255 Jul 16 04:40:15.576872 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.576785 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mlq84" event={"ID":"9870a155-1b41-4300-9007-f16f24f9c3cd","Type":"ContainerDied","Data":"29111863adc7c0d404a2d362b4cb169f4459a1cc0c7fb969be18fd067b7a3ec1"} Jul 16 04:40:15.577283 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.577262 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:15.577357 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.577294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:15.577414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.577402 2585 scope.go:117] "RemoveContainer" containerID="d337107b865a1bf9abce56f8269c87f856516deec9ddb3f78c6719f48bfc92d1" Jul 16 04:40:15.577465 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:15.577419 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:19.577402283 +0000 UTC m=+66.895450821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : configmap references non-existent config key: service-ca.crt Jul 16 04:40:15.577465 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:15.577431 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 04:40:15.577549 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.577479 2585 scope.go:117] "RemoveContainer" containerID="29111863adc7c0d404a2d362b4cb169f4459a1cc0c7fb969be18fd067b7a3ec1" Jul 16 04:40:15.577549 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:15.577490 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:19.577475193 +0000 UTC m=+66.895523745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : secret "router-metrics-certs-default" not found Jul 16 04:40:15.577694 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:15.577671 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mlq84_openshift-console-operator(9870a155-1b41-4300-9007-f16f24f9c3cd)\"" pod="openshift-console-operator/console-operator-7866497494-mlq84" podUID="9870a155-1b41-4300-9007-f16f24f9c3cd" Jul 16 04:40:15.605679 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.605638 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-g9vq4" podStartSLOduration=0.791526709 podStartE2EDuration="3.605624114s" totalImagesPullingTime="2.814097405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:40:15.604053011 +0000 UTC m=+62.922101570" watchObservedRunningTime="2026-07-16 04:40:15.605624114 +0000 UTC m=+62.923672671" Jul 16 04:40:15.779147 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:15.779112 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:15.779314 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:15.779290 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 04:40:15.779374 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:15.779363 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls podName:81dcab13-9c75-445a-a53a-7d6f05491ffd nodeName:}" failed. No retries permitted until 2026-07-16 04:40:19.779345922 +0000 UTC m=+67.097394458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls") pod "cluster-samples-operator-65db858894-c8kw6" (UID: "81dcab13-9c75-445a-a53a-7d6f05491ffd") : secret "samples-operator-tls" not found Jul 16 04:40:16.580264 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.580238 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:40:16.580748 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.580729 2585 scope.go:117] "RemoveContainer" containerID="29111863adc7c0d404a2d362b4cb169f4459a1cc0c7fb969be18fd067b7a3ec1" Jul 16 04:40:16.580960 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:16.580939 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mlq84_openshift-console-operator(9870a155-1b41-4300-9007-f16f24f9c3cd)\"" pod="openshift-console-operator/console-operator-7866497494-mlq84" podUID="9870a155-1b41-4300-9007-f16f24f9c3cd" Jul 16 04:40:16.906266 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.905906 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-46bgf"] Jul 16 04:40:16.910232 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.910211 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:16.917717 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.917694 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 04:40:16.917835 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.917744 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 04:40:16.919162 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.919146 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 04:40:16.919833 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.919818 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 04:40:16.922865 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.922850 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-68dzs\"" Jul 16 04:40:16.945574 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.945551 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-46bgf"] Jul 16 04:40:16.988153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.988125 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbebbdaf-4535-414a-8bce-42ca78476948-data-volume\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:16.988356 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.988290 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:16.988356 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.988335 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:16.988453 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.988366 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l8wm\" (UniqueName: \"kubernetes.io/projected/fbebbdaf-4535-414a-8bce-42ca78476948-kube-api-access-6l8wm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:16.988453 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:16.988391 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbebbdaf-4535-414a-8bce-42ca78476948-crio-socket\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.089663 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.089627 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.089873 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.089671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.089873 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.089697 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l8wm\" (UniqueName: \"kubernetes.io/projected/fbebbdaf-4535-414a-8bce-42ca78476948-kube-api-access-6l8wm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.089873 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.089724 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbebbdaf-4535-414a-8bce-42ca78476948-crio-socket\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.089873 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.089754 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbebbdaf-4535-414a-8bce-42ca78476948-data-volume\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.089873 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.089760 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 04:40:17.089873 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.089790 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 04:40:17.089873 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.089836 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:17.58982221 +0000 UTC m=+64.907870746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : configmap "kube-rbac-proxy" not found Jul 16 04:40:17.089873 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.089859 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:17.589843464 +0000 UTC m=+64.907892007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : secret "insights-runtime-extractor-tls" not found Jul 16 04:40:17.089873 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.089866 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbebbdaf-4535-414a-8bce-42ca78476948-crio-socket\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.090280 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.090145 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbebbdaf-4535-414a-8bce-42ca78476948-data-volume\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.101501 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.101476 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l8wm\" (UniqueName: \"kubernetes.io/projected/fbebbdaf-4535-414a-8bce-42ca78476948-kube-api-access-6l8wm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.596346 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.594501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.596346 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.594566 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:17.596346 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.594803 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 04:40:17.596346 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.594870 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:18.59485086 +0000 UTC m=+65.912899410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : configmap "kube-rbac-proxy" not found Jul 16 04:40:17.596346 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.595358 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 04:40:17.596346 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.595414 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:18.59539835 +0000 UTC m=+65.913446888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : secret "insights-runtime-extractor-tls" not found Jul 16 04:40:17.896549 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.896453 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:40:17.896549 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.896508 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:40:17.898810 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.898784 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w69nk\" (UniqueName: \"kubernetes.io/projected/255f6da7-bab4-4b7c-984f-cd1ddda313e9-kube-api-access-w69nk\") pod \"network-check-target-lgzrj\" (UID: \"255f6da7-bab4-4b7c-984f-cd1ddda313e9\") " pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:40:17.899186 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.899173 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 04:40:17.907035 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.907020 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 04:40:17.907108 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:17.907070 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs podName:42f5e13d-20f3-46ed-ab64-13ce54e5e719 nodeName:}" failed. No retries permitted until 2026-07-16 04:41:21.907056183 +0000 UTC m=+129.225104748 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs") pod "network-metrics-daemon-nsrxl" (UID: "42f5e13d-20f3-46ed-ab64-13ce54e5e719") : secret "metrics-daemon-secret" not found Jul 16 04:40:17.918904 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.918884 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rjh6n\"" Jul 16 04:40:17.927178 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.927158 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:40:17.997715 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:17.997687 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:40:18.001084 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.001046 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 04:40:18.010160 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.010126 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2385ede2-8c97-424e-b560-19011a292b9e-original-pull-secret\") pod \"global-pull-secret-syncer-l6277\" (UID: \"2385ede2-8c97-424e-b560-19011a292b9e\") " pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:40:18.044816 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.044786 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lgzrj"] Jul 16 04:40:18.047665 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:18.047636 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255f6da7_bab4_4b7c_984f_cd1ddda313e9.slice/crio-3329f7f2e42e4810f2fed46c5d4d869b22240a6c224e8092ca4276f3252cf907 WatchSource:0}: Error finding container 3329f7f2e42e4810f2fed46c5d4d869b22240a6c224e8092ca4276f3252cf907: Status 404 returned error can't find the container with id 3329f7f2e42e4810f2fed46c5d4d869b22240a6c224e8092ca4276f3252cf907 Jul 16 04:40:18.230257 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.230183 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6277" Jul 16 04:40:18.350051 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.350021 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6277"] Jul 16 04:40:18.352699 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:18.352669 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2385ede2_8c97_424e_b560_19011a292b9e.slice/crio-6df006a5553bf8cd8540801b39d4c9d73aa8ea874388d8d1ea25a30ee8a63305 WatchSource:0}: Error finding container 6df006a5553bf8cd8540801b39d4c9d73aa8ea874388d8d1ea25a30ee8a63305: Status 404 returned error can't find the container with id 6df006a5553bf8cd8540801b39d4c9d73aa8ea874388d8d1ea25a30ee8a63305 Jul 16 04:40:18.585636 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.585604 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6277" event={"ID":"2385ede2-8c97-424e-b560-19011a292b9e","Type":"ContainerStarted","Data":"6df006a5553bf8cd8540801b39d4c9d73aa8ea874388d8d1ea25a30ee8a63305"} Jul 16 04:40:18.586857 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.586835 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lgzrj" event={"ID":"255f6da7-bab4-4b7c-984f-cd1ddda313e9","Type":"ContainerStarted","Data":"473f7f2f05a6d5837035ad2b1d4e1f4da2f38140b21babd72f0a5b8b6043c302"} Jul 16 04:40:18.586971 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.586873 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lgzrj" event={"ID":"255f6da7-bab4-4b7c-984f-cd1ddda313e9","Type":"ContainerStarted","Data":"3329f7f2e42e4810f2fed46c5d4d869b22240a6c224e8092ca4276f3252cf907"} Jul 16 04:40:18.586971 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.586898 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:40:18.603020 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.602996 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:18.603359 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.603028 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:18.603359 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:18.603113 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 04:40:18.603359 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:18.603141 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 04:40:18.603359 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:18.603157 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:20.603144395 +0000 UTC m=+67.921192938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : configmap "kube-rbac-proxy" not found Jul 16 04:40:18.603359 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:18.603196 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:20.603179522 +0000 UTC m=+67.921228268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : secret "insights-runtime-extractor-tls" not found Jul 16 04:40:18.621124 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.621085 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lgzrj" podStartSLOduration=65.621070656 podStartE2EDuration="1m5.621070656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:40:18.620192444 +0000 UTC m=+65.938241003" watchObservedRunningTime="2026-07-16 04:40:18.621070656 +0000 UTC m=+65.939119192" Jul 16 04:40:18.900717 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:18.900648 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tqh7x_0e63d7c4-a323-4925-a1e2-06690436efe2/dns-node-resolver/0.log" Jul 16 04:40:19.409455 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:19.409416 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:40:19.409641 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:19.409624 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 04:40:19.409701 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:19.409688 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls podName:a1c56c68-9f4a-445b-9739-c33a31e7def0 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:51.409673195 +0000 UTC m=+98.727721737 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls") pod "dns-default-757q2" (UID: "a1c56c68-9f4a-445b-9739-c33a31e7def0") : secret "dns-default-metrics-tls" not found Jul 16 04:40:19.510172 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:19.510137 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:19.510399 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:19.510264 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:19.510399 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:19.510323 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls podName:be0c3d73-85fb-40a9-a2af-001e8231724a nodeName:}" failed. No retries permitted until 2026-07-16 04:40:27.51030759 +0000 UTC m=+74.828356127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-g544c" (UID: "be0c3d73-85fb-40a9-a2af-001e8231724a") : secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:19.610684 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:19.610653 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:19.610684 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:19.610686 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:19.611179 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:19.610841 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:27.610817111 +0000 UTC m=+74.928865666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : configmap references non-existent config key: service-ca.crt Jul 16 04:40:19.611179 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:19.610890 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 04:40:19.611179 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:19.610948 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:27.610932012 +0000 UTC m=+74.928980553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : secret "router-metrics-certs-default" not found Jul 16 04:40:19.703165 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:19.703082 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cgjjz_69a01906-2a04-452a-b44e-9f6971572c5d/node-ca/0.log" Jul 16 04:40:19.812099 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:19.812057 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:19.812294 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:19.812232 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 04:40:19.812353 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:19.812304 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls podName:81dcab13-9c75-445a-a53a-7d6f05491ffd nodeName:}" failed. No retries permitted until 2026-07-16 04:40:27.812286601 +0000 UTC m=+75.130335140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls") pod "cluster-samples-operator-65db858894-c8kw6" (UID: "81dcab13-9c75-445a-a53a-7d6f05491ffd") : secret "samples-operator-tls" not found Jul 16 04:40:20.618693 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:20.618658 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:20.619109 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:20.618713 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:20.619109 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:20.618796 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 04:40:20.619109 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:20.618824 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 04:40:20.619109 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:20.618868 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:24.618848976 +0000 UTC m=+71.936897525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : configmap "kube-rbac-proxy" not found Jul 16 04:40:20.619109 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:20.618881 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:24.618875378 +0000 UTC m=+71.936923914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : secret "insights-runtime-extractor-tls" not found Jul 16 04:40:20.703199 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:20.703169 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qpks8_f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66/serve-healthcheck-canary/0.log" Jul 16 04:40:22.099448 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:22.099415 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:22.099824 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:22.099479 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:22.099824 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:22.099817 2585 scope.go:117] "RemoveContainer" containerID="29111863adc7c0d404a2d362b4cb169f4459a1cc0c7fb969be18fd067b7a3ec1" Jul 16 04:40:22.099978 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:22.099959 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mlq84_openshift-console-operator(9870a155-1b41-4300-9007-f16f24f9c3cd)\"" pod="openshift-console-operator/console-operator-7866497494-mlq84" podUID="9870a155-1b41-4300-9007-f16f24f9c3cd" Jul 16 04:40:22.599192 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:22.599157 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6277" event={"ID":"2385ede2-8c97-424e-b560-19011a292b9e","Type":"ContainerStarted","Data":"fb64240a3cf12440b040c0e7051fded00f57ba16e89d00a742413f0c37836454"} Jul 16 04:40:22.619564 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:22.619469 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l6277" podStartSLOduration=65.631645284 podStartE2EDuration="1m9.619454422s" totalImagesPullingTime="3.987809138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:40:22.618916081 +0000 UTC m=+69.936964638" watchObservedRunningTime="2026-07-16 04:40:22.619454422 +0000 UTC m=+69.937502977" Jul 16 04:40:24.652256 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:24.652211 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:24.652256 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:24.652260 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:24.652685 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:24.652326 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 04:40:24.652685 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:24.652351 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 04:40:24.652685 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:24.652369 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:32.652355621 +0000 UTC m=+79.970404162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : configmap "kube-rbac-proxy" not found Jul 16 04:40:24.652685 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:24.652396 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:32.652384308 +0000 UTC m=+79.970432845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : secret "insights-runtime-extractor-tls" not found Jul 16 04:40:27.578703 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:27.578654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:27.579167 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:27.578840 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:27.579167 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:27.578918 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls podName:be0c3d73-85fb-40a9-a2af-001e8231724a nodeName:}" failed. No retries permitted until 2026-07-16 04:40:43.578898115 +0000 UTC m=+90.896946658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-g544c" (UID: "be0c3d73-85fb-40a9-a2af-001e8231724a") : secret "cluster-monitoring-operator-tls" not found Jul 16 04:40:27.679886 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:27.679854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:27.679886 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:27.679887 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:27.680053 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:27.680030 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle podName:fcf7a78a-b00a-4419-923e-5cbde11f9f85 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:43.680010711 +0000 UTC m=+90.998059249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle") pod "router-default-7b679c4ddd-d64dk" (UID: "fcf7a78a-b00a-4419-923e-5cbde11f9f85") : configmap references non-existent config key: service-ca.crt Jul 16 04:40:27.682178 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:27.682151 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf7a78a-b00a-4419-923e-5cbde11f9f85-metrics-certs\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:27.881810 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:27.881706 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:27.884073 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:27.884049 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcab13-9c75-445a-a53a-7d6f05491ffd-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c8kw6\" (UID: \"81dcab13-9c75-445a-a53a-7d6f05491ffd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:28.036125 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:28.036090 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v5r5s\"" Jul 16 04:40:28.039407 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:28.039394 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" Jul 16 04:40:28.332532 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:28.332503 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6"] Jul 16 04:40:28.617504 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:28.617424 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" event={"ID":"81dcab13-9c75-445a-a53a-7d6f05491ffd","Type":"ContainerStarted","Data":"276dda528f97d67e6a2c6e604d7e13aaaea242b308eca827f4f39487addef5ce"} Jul 16 04:40:30.624789 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:30.624738 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" event={"ID":"81dcab13-9c75-445a-a53a-7d6f05491ffd","Type":"ContainerStarted","Data":"f1b83c9ef79c2c093cae5fb08ff17ad5862e29926d8a99fc7a128a7f3d70bcfe"} Jul 16 04:40:30.625187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:30.624804 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" event={"ID":"81dcab13-9c75-445a-a53a-7d6f05491ffd","Type":"ContainerStarted","Data":"6a523c9b09eda5b23dcf331da965e4d7176e6e529481060726694a2f8e32617c"} Jul 16 04:40:30.685814 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:30.685741 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c8kw6" podStartSLOduration=17.109307382 podStartE2EDuration="18.685725462s" totalImagesPullingTime="1.57641808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:40:30.685642542 +0000 UTC m=+78.003691135" watchObservedRunningTime="2026-07-16 04:40:30.685725462 +0000 UTC m=+78.003774020" Jul 16 04:40:32.719468 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:32.719439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:32.719898 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:32.719475 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:32.719898 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:32.719586 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 04:40:32.719898 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:40:32.719657 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls podName:fbebbdaf-4535-414a-8bce-42ca78476948 nodeName:}" failed. No retries permitted until 2026-07-16 04:40:48.719632466 +0000 UTC m=+96.037681002 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-46bgf" (UID: "fbebbdaf-4535-414a-8bce-42ca78476948") : secret "insights-runtime-extractor-tls" not found Jul 16 04:40:32.720007 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:32.719968 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbebbdaf-4535-414a-8bce-42ca78476948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:37.204403 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:37.204271 2585 scope.go:117] "RemoveContainer" containerID="29111863adc7c0d404a2d362b4cb169f4459a1cc0c7fb969be18fd067b7a3ec1" Jul 16 04:40:37.646817 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:37.646789 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:40:37.646970 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:37.646876 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mlq84" event={"ID":"9870a155-1b41-4300-9007-f16f24f9c3cd","Type":"ContainerStarted","Data":"9a5e9b11bb4d9390f11f6c551c76cc0e7811adc8d2a2228cdd4f1d34b648d722"} Jul 16 04:40:37.647157 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:37.647133 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:37.965517 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:37.965434 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-mlq84" Jul 16 04:40:38.003898 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:38.003851 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-mlq84" podStartSLOduration=25.104298391 podStartE2EDuration="27.003833332s" totalImagesPullingTime="1.899534941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:40:37.687553838 +0000 UTC m=+85.005602415" watchObservedRunningTime="2026-07-16 04:40:38.003833332 +0000 UTC m=+85.321881908" Jul 16 04:40:42.822979 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.822945 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs"] Jul 16 04:40:42.826044 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.826026 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" Jul 16 04:40:42.830097 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.830065 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-w2kzb\"" Jul 16 04:40:42.830216 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.830163 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 04:40:42.830216 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.830175 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 04:40:42.830216 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.830163 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 16 04:40:42.830331 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.830274 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 04:40:42.837114 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.837093 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs"] Jul 16 04:40:42.897522 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.897495 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4f48\" (UniqueName: \"kubernetes.io/projected/bc544c4e-421c-4cf4-a620-6cd1175f2144-kube-api-access-p4f48\") pod \"managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs\" (UID: \"bc544c4e-421c-4cf4-a620-6cd1175f2144\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" Jul 16 04:40:42.897522 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.897526 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc544c4e-421c-4cf4-a620-6cd1175f2144-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs\" (UID: \"bc544c4e-421c-4cf4-a620-6cd1175f2144\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" Jul 16 04:40:42.941439 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.941409 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-z7gd5"] Jul 16 04:40:42.944309 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.944293 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-z7gd5" Jul 16 04:40:42.951252 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.951229 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wtddp\"" Jul 16 04:40:42.951408 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.951391 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 04:40:42.951474 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.951395 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 04:40:42.966188 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.966170 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-z7gd5"] Jul 16 04:40:42.998660 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.998636 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f48\" (UniqueName: \"kubernetes.io/projected/bc544c4e-421c-4cf4-a620-6cd1175f2144-kube-api-access-p4f48\") pod \"managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs\" (UID: \"bc544c4e-421c-4cf4-a620-6cd1175f2144\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" Jul 16 04:40:42.998802 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:42.998667 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc544c4e-421c-4cf4-a620-6cd1175f2144-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs\" (UID: \"bc544c4e-421c-4cf4-a620-6cd1175f2144\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" Jul 16 04:40:43.001122 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.001100 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc544c4e-421c-4cf4-a620-6cd1175f2144-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs\" (UID: \"bc544c4e-421c-4cf4-a620-6cd1175f2144\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" Jul 16 04:40:43.034448 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.034415 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4f48\" (UniqueName: \"kubernetes.io/projected/bc544c4e-421c-4cf4-a620-6cd1175f2144-kube-api-access-p4f48\") pod \"managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs\" (UID: \"bc544c4e-421c-4cf4-a620-6cd1175f2144\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" Jul 16 04:40:43.060887 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.060863 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57b9c5599-rzggx"] Jul 16 04:40:43.063816 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.063800 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.069104 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.069078 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xvsxh\"" Jul 16 04:40:43.069203 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.069151 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 04:40:43.071158 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.071142 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 04:40:43.071932 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.071909 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 04:40:43.079257 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.079203 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 04:40:43.100086 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.100066 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnx98\" (UniqueName: \"kubernetes.io/projected/81762e9a-efa8-4042-9f35-3458db40e2d8-kube-api-access-lnx98\") pod \"downloads-68c9cbb497-z7gd5\" (UID: \"81762e9a-efa8-4042-9f35-3458db40e2d8\") " pod="openshift-console/downloads-68c9cbb497-z7gd5" Jul 16 04:40:43.104269 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.104251 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b9c5599-rzggx"] Jul 16 04:40:43.145007 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.144981 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" Jul 16 04:40:43.201131 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.201100 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19716920-78d2-4a9c-a3ca-6f0474e9b664-bound-sa-token\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.201387 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.201349 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mv4\" (UniqueName: \"kubernetes.io/projected/19716920-78d2-4a9c-a3ca-6f0474e9b664-kube-api-access-b5mv4\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.201544 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.201527 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19716920-78d2-4a9c-a3ca-6f0474e9b664-registry-tls\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.201753 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.201715 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19716920-78d2-4a9c-a3ca-6f0474e9b664-trusted-ca\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.201850 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.201807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19716920-78d2-4a9c-a3ca-6f0474e9b664-ca-trust-extracted\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.201906 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.201874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnx98\" (UniqueName: \"kubernetes.io/projected/81762e9a-efa8-4042-9f35-3458db40e2d8-kube-api-access-lnx98\") pod \"downloads-68c9cbb497-z7gd5\" (UID: \"81762e9a-efa8-4042-9f35-3458db40e2d8\") " pod="openshift-console/downloads-68c9cbb497-z7gd5" Jul 16 04:40:43.201956 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.201938 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19716920-78d2-4a9c-a3ca-6f0474e9b664-installation-pull-secrets\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.202011 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.201983 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19716920-78d2-4a9c-a3ca-6f0474e9b664-image-registry-private-configuration\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.202063 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.202021 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19716920-78d2-4a9c-a3ca-6f0474e9b664-registry-certificates\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.217150 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.217119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnx98\" (UniqueName: \"kubernetes.io/projected/81762e9a-efa8-4042-9f35-3458db40e2d8-kube-api-access-lnx98\") pod \"downloads-68c9cbb497-z7gd5\" (UID: \"81762e9a-efa8-4042-9f35-3458db40e2d8\") " pod="openshift-console/downloads-68c9cbb497-z7gd5" Jul 16 04:40:43.252643 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.252616 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-z7gd5" Jul 16 04:40:43.301169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.301140 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs"] Jul 16 04:40:43.303518 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.303067 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19716920-78d2-4a9c-a3ca-6f0474e9b664-installation-pull-secrets\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.303518 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.303118 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19716920-78d2-4a9c-a3ca-6f0474e9b664-image-registry-private-configuration\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.303518 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.303155 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19716920-78d2-4a9c-a3ca-6f0474e9b664-registry-certificates\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.303518 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.303186 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19716920-78d2-4a9c-a3ca-6f0474e9b664-bound-sa-token\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.303518 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.303210 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mv4\" (UniqueName: \"kubernetes.io/projected/19716920-78d2-4a9c-a3ca-6f0474e9b664-kube-api-access-b5mv4\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.303518 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.303239 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19716920-78d2-4a9c-a3ca-6f0474e9b664-registry-tls\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.303518 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.303262 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19716920-78d2-4a9c-a3ca-6f0474e9b664-trusted-ca\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.303518 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.303300 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19716920-78d2-4a9c-a3ca-6f0474e9b664-ca-trust-extracted\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.303988 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.303665 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19716920-78d2-4a9c-a3ca-6f0474e9b664-ca-trust-extracted\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.304338 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.304318 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19716920-78d2-4a9c-a3ca-6f0474e9b664-registry-certificates\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.304909 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.304882 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19716920-78d2-4a9c-a3ca-6f0474e9b664-trusted-ca\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.305222 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:43.305197 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc544c4e_421c_4cf4_a620_6cd1175f2144.slice/crio-af6c18df6d028f5d918f7470e203a4eac2e8c7d00b6734a869f4ac1d7cc5bcd8 WatchSource:0}: Error finding container af6c18df6d028f5d918f7470e203a4eac2e8c7d00b6734a869f4ac1d7cc5bcd8: Status 404 returned error can't find the container with id af6c18df6d028f5d918f7470e203a4eac2e8c7d00b6734a869f4ac1d7cc5bcd8 Jul 16 04:40:43.306626 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.306397 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19716920-78d2-4a9c-a3ca-6f0474e9b664-registry-tls\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.306626 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.306592 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19716920-78d2-4a9c-a3ca-6f0474e9b664-installation-pull-secrets\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.308115 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.307086 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19716920-78d2-4a9c-a3ca-6f0474e9b664-image-registry-private-configuration\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.323584 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.323564 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19716920-78d2-4a9c-a3ca-6f0474e9b664-bound-sa-token\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.334389 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.334360 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mv4\" (UniqueName: \"kubernetes.io/projected/19716920-78d2-4a9c-a3ca-6f0474e9b664-kube-api-access-b5mv4\") pod \"image-registry-57b9c5599-rzggx\" (UID: \"19716920-78d2-4a9c-a3ca-6f0474e9b664\") " pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.371558 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.371534 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.385551 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.385532 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-z7gd5"] Jul 16 04:40:43.387481 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:43.387461 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81762e9a_efa8_4042_9f35_3458db40e2d8.slice/crio-c6d86638b1702735ae5191e487ec076ee7f71d462edec07be2df8706ffe558a5 WatchSource:0}: Error finding container c6d86638b1702735ae5191e487ec076ee7f71d462edec07be2df8706ffe558a5: Status 404 returned error can't find the container with id c6d86638b1702735ae5191e487ec076ee7f71d462edec07be2df8706ffe558a5 Jul 16 04:40:43.506908 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.506870 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b9c5599-rzggx"] Jul 16 04:40:43.510988 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:43.510965 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19716920_78d2_4a9c_a3ca_6f0474e9b664.slice/crio-04aeeac9f9343f3b8b6f977060e4f25ff1a834f520e52465d22c09de6bd23f85 WatchSource:0}: Error finding container 04aeeac9f9343f3b8b6f977060e4f25ff1a834f520e52465d22c09de6bd23f85: Status 404 returned error can't find the container with id 04aeeac9f9343f3b8b6f977060e4f25ff1a834f520e52465d22c09de6bd23f85 Jul 16 04:40:43.606224 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.606151 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:43.608498 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.608469 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/be0c3d73-85fb-40a9-a2af-001e8231724a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-g544c\" (UID: \"be0c3d73-85fb-40a9-a2af-001e8231724a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:43.664575 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.664547 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" event={"ID":"bc544c4e-421c-4cf4-a620-6cd1175f2144","Type":"ContainerStarted","Data":"af6c18df6d028f5d918f7470e203a4eac2e8c7d00b6734a869f4ac1d7cc5bcd8"} Jul 16 04:40:43.665692 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.665658 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-z7gd5" event={"ID":"81762e9a-efa8-4042-9f35-3458db40e2d8","Type":"ContainerStarted","Data":"c6d86638b1702735ae5191e487ec076ee7f71d462edec07be2df8706ffe558a5"} Jul 16 04:40:43.667052 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.667032 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b9c5599-rzggx" event={"ID":"19716920-78d2-4a9c-a3ca-6f0474e9b664","Type":"ContainerStarted","Data":"09376426704c2aec3bd5274378aa9dd14b3102611a83294980744f2bfbd6ce96"} Jul 16 04:40:43.667163 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.667065 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b9c5599-rzggx" event={"ID":"19716920-78d2-4a9c-a3ca-6f0474e9b664","Type":"ContainerStarted","Data":"04aeeac9f9343f3b8b6f977060e4f25ff1a834f520e52465d22c09de6bd23f85"} Jul 16 04:40:43.667163 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.667155 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:40:43.707329 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.707293 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:43.708148 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.708119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf7a78a-b00a-4419-923e-5cbde11f9f85-service-ca-bundle\") pod \"router-default-7b679c4ddd-d64dk\" (UID: \"fcf7a78a-b00a-4419-923e-5cbde11f9f85\") " pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:43.723286 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.723233 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57b9c5599-rzggx" podStartSLOduration=0.723215268 podStartE2EDuration="723.215268ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:40:43.71793421 +0000 UTC m=+91.035982778" watchObservedRunningTime="2026-07-16 04:40:43.723215268 +0000 UTC m=+91.041263827" Jul 16 04:40:43.752844 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.752817 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hprbf\"" Jul 16 04:40:43.762387 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.762370 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:43.895374 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.895313 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-w2tj7\"" Jul 16 04:40:43.900428 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.900374 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" Jul 16 04:40:43.932474 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:43.932440 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b679c4ddd-d64dk"] Jul 16 04:40:44.058565 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:44.058531 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0c3d73_85fb_40a9_a2af_001e8231724a.slice/crio-80db6826b8dd4fa67f64ce4677f2e7aed662a40492c1678e58ffacbad9f8af75 WatchSource:0}: Error finding container 80db6826b8dd4fa67f64ce4677f2e7aed662a40492c1678e58ffacbad9f8af75: Status 404 returned error can't find the container with id 80db6826b8dd4fa67f64ce4677f2e7aed662a40492c1678e58ffacbad9f8af75 Jul 16 04:40:44.059353 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:44.059326 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c"] Jul 16 04:40:44.676241 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:44.676175 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" event={"ID":"be0c3d73-85fb-40a9-a2af-001e8231724a","Type":"ContainerStarted","Data":"80db6826b8dd4fa67f64ce4677f2e7aed662a40492c1678e58ffacbad9f8af75"} Jul 16 04:40:44.678512 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:44.678481 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b679c4ddd-d64dk" event={"ID":"fcf7a78a-b00a-4419-923e-5cbde11f9f85","Type":"ContainerStarted","Data":"65eec5bcfafb4af4a87990976d51b3443aaf3b84e8f9f7db57063af2c683d498"} Jul 16 04:40:44.678645 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:44.678527 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b679c4ddd-d64dk" event={"ID":"fcf7a78a-b00a-4419-923e-5cbde11f9f85","Type":"ContainerStarted","Data":"c5b72c3612dd24ae38aedd9e088948d9f223af87673587170527cd6c55e337b2"} Jul 16 04:40:44.713117 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:44.713037 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7b679c4ddd-d64dk" podStartSLOduration=33.71301953 podStartE2EDuration="33.71301953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:40:44.71221956 +0000 UTC m=+92.030268140" watchObservedRunningTime="2026-07-16 04:40:44.71301953 +0000 UTC m=+92.031068089" Jul 16 04:40:44.762967 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:44.762934 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:44.765863 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:44.765838 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:45.683970 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:45.683537 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:45.685263 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:45.685231 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b679c4ddd-d64dk" Jul 16 04:40:46.690831 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:46.690791 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" event={"ID":"be0c3d73-85fb-40a9-a2af-001e8231724a","Type":"ContainerStarted","Data":"983076bbf7db0cf08fd988aa5c63861292358808d47b512c9d4012eb176eedc0"} Jul 16 04:40:46.731385 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:46.731334 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-g544c" podStartSLOduration=33.887645421 podStartE2EDuration="35.731320035s" totalImagesPullingTime="1.843674614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:40:46.730708673 +0000 UTC m=+94.048757223" watchObservedRunningTime="2026-07-16 04:40:46.731320035 +0000 UTC m=+94.049368592" Jul 16 04:40:48.749819 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:48.749776 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:48.752281 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:48.752254 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbebbdaf-4535-414a-8bce-42ca78476948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-46bgf\" (UID: \"fbebbdaf-4535-414a-8bce-42ca78476948\") " pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:49.018878 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:49.018790 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-46bgf" Jul 16 04:40:49.202678 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:49.202551 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-46bgf"] Jul 16 04:40:49.592067 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:49.592042 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lgzrj" Jul 16 04:40:49.704558 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:49.704521 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46bgf" event={"ID":"fbebbdaf-4535-414a-8bce-42ca78476948","Type":"ContainerStarted","Data":"719626e7def6c76036ad5e4164c59de09b980d895a862def04547b9952dd2c56"} Jul 16 04:40:50.931908 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:50.931860 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-4s9jr"] Jul 16 04:40:50.935111 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:50.934840 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:50.939159 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:50.939116 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 04:40:50.939269 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:50.939163 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qkxxx\"" Jul 16 04:40:50.939408 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:50.939390 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 04:40:50.940419 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:50.939949 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 04:40:50.951512 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:50.951380 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-4s9jr"] Jul 16 04:40:51.069875 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.069839 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6544a76c-1633-4df6-a536-3e5318a41cc2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.070223 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.069907 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6544a76c-1633-4df6-a536-3e5318a41cc2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.070223 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.069998 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwr2\" (UniqueName: \"kubernetes.io/projected/6544a76c-1633-4df6-a536-3e5318a41cc2-kube-api-access-rhwr2\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.070223 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.070030 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6544a76c-1633-4df6-a536-3e5318a41cc2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.171403 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.171362 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwr2\" (UniqueName: \"kubernetes.io/projected/6544a76c-1633-4df6-a536-3e5318a41cc2-kube-api-access-rhwr2\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.171560 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.171416 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6544a76c-1633-4df6-a536-3e5318a41cc2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.171560 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.171451 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6544a76c-1633-4df6-a536-3e5318a41cc2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.171560 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.171492 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6544a76c-1633-4df6-a536-3e5318a41cc2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.172280 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.172250 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6544a76c-1633-4df6-a536-3e5318a41cc2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.174618 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.174455 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6544a76c-1633-4df6-a536-3e5318a41cc2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.175068 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.175046 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6544a76c-1633-4df6-a536-3e5318a41cc2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.187151 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.187074 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwr2\" (UniqueName: \"kubernetes.io/projected/6544a76c-1633-4df6-a536-3e5318a41cc2-kube-api-access-rhwr2\") pod \"prometheus-operator-64785b6675-4s9jr\" (UID: \"6544a76c-1633-4df6-a536-3e5318a41cc2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.247011 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.246976 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" Jul 16 04:40:51.473962 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.473885 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:40:51.477520 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.477493 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1c56c68-9f4a-445b-9739-c33a31e7def0-metrics-tls\") pod \"dns-default-757q2\" (UID: \"a1c56c68-9f4a-445b-9739-c33a31e7def0\") " pod="openshift-dns/dns-default-757q2" Jul 16 04:40:51.488184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.488158 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pmfl6\"" Jul 16 04:40:51.495420 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.495395 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-757q2" Jul 16 04:40:51.950749 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.950654 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-4s9jr"] Jul 16 04:40:51.955556 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:51.955523 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6544a76c_1633_4df6_a536_3e5318a41cc2.slice/crio-49a7104b5dbf8da44c584bbdbcf3625bf6568bf8e44cb8961de1e3304b83463a WatchSource:0}: Error finding container 49a7104b5dbf8da44c584bbdbcf3625bf6568bf8e44cb8961de1e3304b83463a: Status 404 returned error can't find the container with id 49a7104b5dbf8da44c584bbdbcf3625bf6568bf8e44cb8961de1e3304b83463a Jul 16 04:40:51.983679 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:51.983615 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-757q2"] Jul 16 04:40:51.988757 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:40:51.988728 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c56c68_9f4a_445b_9739_c33a31e7def0.slice/crio-bbe0a5c5606e52eb9343ca14ad3f9f7bf9bdd4606002d970fbe18314836304b3 WatchSource:0}: Error finding container bbe0a5c5606e52eb9343ca14ad3f9f7bf9bdd4606002d970fbe18314836304b3: Status 404 returned error can't find the container with id bbe0a5c5606e52eb9343ca14ad3f9f7bf9bdd4606002d970fbe18314836304b3 Jul 16 04:40:52.723375 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:52.723340 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" event={"ID":"bc544c4e-421c-4cf4-a620-6cd1175f2144","Type":"ContainerStarted","Data":"66f5e7dd6e2ac74866be9b66d933618348ca3e3f48e933511e6c7ecf9153e9e3"} Jul 16 04:40:52.724722 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:52.724692 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-757q2" event={"ID":"a1c56c68-9f4a-445b-9739-c33a31e7def0","Type":"ContainerStarted","Data":"bbe0a5c5606e52eb9343ca14ad3f9f7bf9bdd4606002d970fbe18314836304b3"} Jul 16 04:40:52.726080 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:52.726057 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" event={"ID":"6544a76c-1633-4df6-a536-3e5318a41cc2","Type":"ContainerStarted","Data":"49a7104b5dbf8da44c584bbdbcf3625bf6568bf8e44cb8961de1e3304b83463a"} Jul 16 04:40:52.744370 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:40:52.744232 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d7dc65fd8-26dcs" podStartSLOduration=2.247227089 podStartE2EDuration="10.744214375s" totalImagesPullingTime="8.496987286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:40:52.743593835 +0000 UTC m=+100.061642390" watchObservedRunningTime="2026-07-16 04:40:52.744214375 +0000 UTC m=+100.062262934" Jul 16 04:41:03.768256 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.768215 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46bgf" event={"ID":"fbebbdaf-4535-414a-8bce-42ca78476948","Type":"ContainerStarted","Data":"c3229714770b08710d6b9d3fbcd84043a1dc11eeeccfe3a6768bb632477a7094"} Jul 16 04:41:03.770964 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.770933 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-757q2" event={"ID":"a1c56c68-9f4a-445b-9739-c33a31e7def0","Type":"ContainerStarted","Data":"23eace144228ec91ac6bcb4e2807c8b8a611339d7665c3cb0ec33196e46b98a7"} Jul 16 04:41:03.771099 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.770989 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-757q2" event={"ID":"a1c56c68-9f4a-445b-9739-c33a31e7def0","Type":"ContainerStarted","Data":"ff7a72b842f465c47d91b3f4894cc15267df3a8299be38e6bd14f2bbf4d0471e"} Jul 16 04:41:03.771893 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.771840 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-757q2" Jul 16 04:41:03.774948 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.774865 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" event={"ID":"6544a76c-1633-4df6-a536-3e5318a41cc2","Type":"ContainerStarted","Data":"8a15c31ee1265fe044c3290190930ee55931b7699be06997c6860fca9e714373"} Jul 16 04:41:03.774948 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.774913 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" event={"ID":"6544a76c-1633-4df6-a536-3e5318a41cc2","Type":"ContainerStarted","Data":"3c56c9c7e576a56b72d0c5af5fcef44e24a3cd207aaa4d4a96228e74557c039d"} Jul 16 04:41:03.778562 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.777565 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-z7gd5" event={"ID":"81762e9a-efa8-4042-9f35-3458db40e2d8","Type":"ContainerStarted","Data":"96bba60a4baf5a0c101f2d309f027e479b8a3b2ba8f996ed46e9e42dbfdedd80"} Jul 16 04:41:03.778562 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.778472 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-z7gd5" Jul 16 04:41:03.791107 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.791083 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-z7gd5" Jul 16 04:41:03.806203 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.806147 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-757q2" podStartSLOduration=65.946925656 podStartE2EDuration="1m16.806129943s" totalImagesPullingTime="10.859204287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:41:03.80309688 +0000 UTC m=+111.121145456" watchObservedRunningTime="2026-07-16 04:41:03.806129943 +0000 UTC m=+111.124178700" Jul 16 04:41:03.861660 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:03.861341 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-z7gd5" podStartSLOduration=2.409178277 podStartE2EDuration="21.861325773s" totalImagesPullingTime="19.452147496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:41:03.8604668 +0000 UTC m=+111.178515362" watchObservedRunningTime="2026-07-16 04:41:03.861325773 +0000 UTC m=+111.179374333" Jul 16 04:41:04.684242 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:04.684209 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57b9c5599-rzggx" Jul 16 04:41:04.725116 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:04.724625 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-4s9jr" podStartSLOduration=3.829976296 podStartE2EDuration="14.724600453s" totalImagesPullingTime="10.894624157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:41:03.918043426 +0000 UTC m=+111.236092006" watchObservedRunningTime="2026-07-16 04:41:04.724600453 +0000 UTC m=+112.042649008" Jul 16 04:41:07.181659 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.181627 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gn9lh"] Jul 16 04:41:07.205782 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.205735 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.222247 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.222205 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kdsbv\"" Jul 16 04:41:07.223665 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.222650 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 04:41:07.223665 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.223052 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 04:41:07.223665 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.223492 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 04:41:07.311382 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.311342 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9800c6e8-c167-4cf7-9da2-c09d37fb2872-metrics-client-ca\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.311546 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.311397 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9800c6e8-c167-4cf7-9da2-c09d37fb2872-root\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.311546 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.311423 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.311546 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.311449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-accelerators-collector-config\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.311546 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.311491 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-wtmp\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.311546 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.311512 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9800c6e8-c167-4cf7-9da2-c09d37fb2872-sys\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.311546 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.311537 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-textfile\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.311897 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.311559 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-tls\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.311897 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.311637 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnh5f\" (UniqueName: \"kubernetes.io/projected/9800c6e8-c167-4cf7-9da2-c09d37fb2872-kube-api-access-nnh5f\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.412405 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412365 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnh5f\" (UniqueName: \"kubernetes.io/projected/9800c6e8-c167-4cf7-9da2-c09d37fb2872-kube-api-access-nnh5f\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.412405 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412412 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9800c6e8-c167-4cf7-9da2-c09d37fb2872-metrics-client-ca\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.412629 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412435 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9800c6e8-c167-4cf7-9da2-c09d37fb2872-root\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.412629 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412483 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9800c6e8-c167-4cf7-9da2-c09d37fb2872-root\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.412629 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412508 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.412629 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412531 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-accelerators-collector-config\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.412629 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412567 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-wtmp\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.412629 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9800c6e8-c167-4cf7-9da2-c09d37fb2872-sys\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.412629 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412618 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-textfile\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.413004 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412640 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-tls\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.413004 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412672 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9800c6e8-c167-4cf7-9da2-c09d37fb2872-sys\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.413004 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412733 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-wtmp\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.413004 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.412933 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-textfile\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.413184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.413095 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-accelerators-collector-config\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.413184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.413100 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9800c6e8-c167-4cf7-9da2-c09d37fb2872-metrics-client-ca\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.415163 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.415115 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.415281 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.415197 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9800c6e8-c167-4cf7-9da2-c09d37fb2872-node-exporter-tls\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.449555 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.449494 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnh5f\" (UniqueName: \"kubernetes.io/projected/9800c6e8-c167-4cf7-9da2-c09d37fb2872-kube-api-access-nnh5f\") pod \"node-exporter-gn9lh\" (UID: \"9800c6e8-c167-4cf7-9da2-c09d37fb2872\") " pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.517392 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.517362 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gn9lh" Jul 16 04:41:07.679609 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:41:07.679573 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9800c6e8_c167_4cf7_9da2_c09d37fb2872.slice/crio-8e077f04a982163a60dde1884d60f07fdbce7be9fd94db1673258461b9ab856f WatchSource:0}: Error finding container 8e077f04a982163a60dde1884d60f07fdbce7be9fd94db1673258461b9ab856f: Status 404 returned error can't find the container with id 8e077f04a982163a60dde1884d60f07fdbce7be9fd94db1673258461b9ab856f Jul 16 04:41:07.801094 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.801048 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46bgf" event={"ID":"fbebbdaf-4535-414a-8bce-42ca78476948","Type":"ContainerStarted","Data":"aea7ff8762dc5be2b90ea45611b42ce11924c9579c12805933c47eb58f24ccb4"} Jul 16 04:41:07.802464 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:07.802426 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gn9lh" event={"ID":"9800c6e8-c167-4cf7-9da2-c09d37fb2872","Type":"ContainerStarted","Data":"8e077f04a982163a60dde1884d60f07fdbce7be9fd94db1673258461b9ab856f"} Jul 16 04:41:09.814681 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:09.814644 2585 generic.go:356] "Generic (PLEG): container finished" podID="9800c6e8-c167-4cf7-9da2-c09d37fb2872" containerID="fea1867cf166ac2a5c89424f269508b95399262eee3ee48277c9e4f4873b37e8" exitCode=0 Jul 16 04:41:09.815225 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:09.814716 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gn9lh" event={"ID":"9800c6e8-c167-4cf7-9da2-c09d37fb2872","Type":"ContainerDied","Data":"fea1867cf166ac2a5c89424f269508b95399262eee3ee48277c9e4f4873b37e8"} Jul 16 04:41:10.821722 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:10.821684 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gn9lh" event={"ID":"9800c6e8-c167-4cf7-9da2-c09d37fb2872","Type":"ContainerStarted","Data":"8c533095c3a8dc9708fb0d6b5352876a5f4dbe9c321efee42da4127a1b9f679a"} Jul 16 04:41:10.822195 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:10.821741 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gn9lh" event={"ID":"9800c6e8-c167-4cf7-9da2-c09d37fb2872","Type":"ContainerStarted","Data":"46ccb5f4723e7ffc127e77f1c567fd51080f2e8e2d45b56a65cd497cab4f2633"} Jul 16 04:41:10.875106 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:10.875046 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gn9lh" podStartSLOduration=2.130225288 podStartE2EDuration="3.87502555s" totalImagesPullingTime="1.650010339s" totalInitContainerRuntime="94.789923ms" isStatefulPod=true podCreationTimestamp="2026-07-16 04:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:41:10.873942409 +0000 UTC m=+118.191990969" watchObservedRunningTime="2026-07-16 04:41:10.87502555 +0000 UTC m=+118.193074109" Jul 16 04:41:11.446841 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.446805 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-924nw"] Jul 16 04:41:11.586956 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.586856 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-924nw"] Jul 16 04:41:11.587161 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.586992 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:11.592678 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.592492 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 16 04:41:11.592678 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.592593 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rch79\"" Jul 16 04:41:11.650912 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.650871 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlwr\" (UniqueName: \"kubernetes.io/projected/2818bc5b-8877-4715-b054-3ad6065eb6ed-kube-api-access-bzlwr\") pod \"monitoring-plugin-5f68df7f58-924nw\" (UID: \"2818bc5b-8877-4715-b054-3ad6065eb6ed\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:11.651091 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.650924 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2818bc5b-8877-4715-b054-3ad6065eb6ed-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-924nw\" (UID: \"2818bc5b-8877-4715-b054-3ad6065eb6ed\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:11.752656 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.752107 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlwr\" (UniqueName: \"kubernetes.io/projected/2818bc5b-8877-4715-b054-3ad6065eb6ed-kube-api-access-bzlwr\") pod \"monitoring-plugin-5f68df7f58-924nw\" (UID: \"2818bc5b-8877-4715-b054-3ad6065eb6ed\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:11.752656 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.752160 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2818bc5b-8877-4715-b054-3ad6065eb6ed-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-924nw\" (UID: \"2818bc5b-8877-4715-b054-3ad6065eb6ed\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:11.755115 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.755062 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2818bc5b-8877-4715-b054-3ad6065eb6ed-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-924nw\" (UID: \"2818bc5b-8877-4715-b054-3ad6065eb6ed\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:11.786401 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.786355 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlwr\" (UniqueName: \"kubernetes.io/projected/2818bc5b-8877-4715-b054-3ad6065eb6ed-kube-api-access-bzlwr\") pod \"monitoring-plugin-5f68df7f58-924nw\" (UID: \"2818bc5b-8877-4715-b054-3ad6065eb6ed\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:11.899034 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:11.899000 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:12.421560 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:12.421534 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-924nw"] Jul 16 04:41:12.529603 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:41:12.529528 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2818bc5b_8877_4715_b054_3ad6065eb6ed.slice/crio-f57c5be0a7db9b1c86bb11ba2d16b798d0a5ca9c4baa0e3a167b8ac2cfca4437 WatchSource:0}: Error finding container f57c5be0a7db9b1c86bb11ba2d16b798d0a5ca9c4baa0e3a167b8ac2cfca4437: Status 404 returned error can't find the container with id f57c5be0a7db9b1c86bb11ba2d16b798d0a5ca9c4baa0e3a167b8ac2cfca4437 Jul 16 04:41:12.831958 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:12.831923 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-46bgf" event={"ID":"fbebbdaf-4535-414a-8bce-42ca78476948","Type":"ContainerStarted","Data":"5808ba07d6d87c7d24e97749ba614bbbe6c5c17aab7f5715da6240ddedd0292a"} Jul 16 04:41:12.833083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:12.833059 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" event={"ID":"2818bc5b-8877-4715-b054-3ad6065eb6ed","Type":"ContainerStarted","Data":"f57c5be0a7db9b1c86bb11ba2d16b798d0a5ca9c4baa0e3a167b8ac2cfca4437"} Jul 16 04:41:14.060543 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.060492 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-46bgf" podStartSLOduration=34.906949182 podStartE2EDuration="58.060475923s" totalImagesPullingTime="23.153526741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 04:40:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:41:12.881174514 +0000 UTC m=+120.199223071" watchObservedRunningTime="2026-07-16 04:41:14.060475923 +0000 UTC m=+121.378524483" Jul 16 04:41:14.062033 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.061998 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 04:41:14.077903 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.077870 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.084517 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:41:14.084054 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-k8s-thanos-sidecar-tls\" is forbidden: User \"system:node:ip-10-0-140-3.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-3.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" Jul 16 04:41:14.084898 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.084874 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 04:41:14.086613 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.086588 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 04:41:14.086887 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.086865 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fb5tv7d79usbk\"" Jul 16 04:41:14.086983 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.086902 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8pkrn\"" Jul 16 04:41:14.087639 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.087626 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 04:41:14.087687 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.087645 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 04:41:14.094476 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.094459 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 04:41:14.094543 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.094492 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 04:41:14.094591 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.094559 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 04:41:14.094792 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.094777 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 04:41:14.096909 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.096887 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 04:41:14.096989 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.096923 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 04:41:14.114216 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.114183 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 04:41:14.120279 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.120263 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 04:41:14.125640 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.125624 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 04:41:14.127877 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.127859 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 04:41:14.172676 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.172645 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 04:41:14.175673 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175612 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.175673 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175653 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.175845 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175684 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.175845 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175800 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.175845 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175827 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.175969 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175865 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.175969 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175916 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.175969 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175938 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkdc\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-kube-api-access-fvkdc\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.175969 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175957 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176137 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176137 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.175990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176137 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.176073 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176249 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.176161 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176249 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.176193 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176249 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.176222 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176249 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.176247 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176378 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.176285 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176378 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.176321 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176378 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.176348 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.176378 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.176370 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.277645 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277612 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.277828 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.277828 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277674 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkdc\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-kube-api-access-fvkdc\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.277828 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.277828 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277708 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.277828 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.277828 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277761 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.277828 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277808 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277843 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277864 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277890 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277913 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277939 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277963 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.277987 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.278017 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.278042 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.278069 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.278101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278200 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.278125 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278756 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.278273 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278756 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.278360 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278756 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.278468 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.278756 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.278699 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.279544 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.279217 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.280916 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.280865 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.281308 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.281285 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.281741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.281713 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.281960 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.281936 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.282585 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.282562 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.282676 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.282639 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.282733 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.282681 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.282858 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.282821 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.283012 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.282994 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.283102 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.283068 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.283102 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.283085 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.283694 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.283674 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.283806 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.283748 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.289973 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.289953 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkdc\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-kube-api-access-fvkdc\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:14.792743 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.792717 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-757q2" Jul 16 04:41:14.842315 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.842282 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" event={"ID":"2818bc5b-8877-4715-b054-3ad6065eb6ed","Type":"ContainerStarted","Data":"141218843a9c2572e02359a19547ca69ddb5950ab4e3b5480922770ba1f79373"} Jul 16 04:41:14.842512 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.842494 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:14.846868 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.846846 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" Jul 16 04:41:14.903965 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:14.903922 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-924nw" podStartSLOduration=2.093438532 podStartE2EDuration="3.903907541s" totalImagesPullingTime="1.810469009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:41:14.902677144 +0000 UTC m=+122.220725703" watchObservedRunningTime="2026-07-16 04:41:14.903907541 +0000 UTC m=+122.221956099" Jul 16 04:41:15.030883 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:15.030852 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 04:41:15.040141 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:15.040114 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:15.288236 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:15.288205 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:15.475004 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:15.471818 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 04:41:15.847111 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:15.847078 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerStarted","Data":"442192912c818e3af56b10e99444ce0de5b7d959f0432747f33724fbc3448ab6"} Jul 16 04:41:17.855750 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:17.855710 2585 generic.go:356] "Generic (PLEG): container finished" podID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerID="aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929" exitCode=0 Jul 16 04:41:17.856202 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:17.855792 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerDied","Data":"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929"} Jul 16 04:41:21.873756 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:21.873725 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerStarted","Data":"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd"} Jul 16 04:41:21.874127 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:21.873810 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerStarted","Data":"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f"} Jul 16 04:41:21.950971 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:21.950895 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:41:21.953381 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:21.953342 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42f5e13d-20f3-46ed-ab64-13ce54e5e719-metrics-certs\") pod \"network-metrics-daemon-nsrxl\" (UID: \"42f5e13d-20f3-46ed-ab64-13ce54e5e719\") " pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:41:22.146309 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:22.146100 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6n555\"" Jul 16 04:41:22.154481 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:22.154451 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrxl" Jul 16 04:41:22.364357 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:22.364332 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nsrxl"] Jul 16 04:41:22.367308 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:41:22.367276 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f5e13d_20f3_46ed_ab64_13ce54e5e719.slice/crio-60a0b32b6a4b2c07f55abd45b2b23708a3090d517cdb9c4d036a65cdaf7f34ce WatchSource:0}: Error finding container 60a0b32b6a4b2c07f55abd45b2b23708a3090d517cdb9c4d036a65cdaf7f34ce: Status 404 returned error can't find the container with id 60a0b32b6a4b2c07f55abd45b2b23708a3090d517cdb9c4d036a65cdaf7f34ce Jul 16 04:41:22.880503 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:22.880464 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nsrxl" event={"ID":"42f5e13d-20f3-46ed-ab64-13ce54e5e719","Type":"ContainerStarted","Data":"60a0b32b6a4b2c07f55abd45b2b23708a3090d517cdb9c4d036a65cdaf7f34ce"} Jul 16 04:41:24.889628 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:24.889589 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nsrxl" event={"ID":"42f5e13d-20f3-46ed-ab64-13ce54e5e719","Type":"ContainerStarted","Data":"7405f1552957506067ddc7c0b27a19d42d26ae37b7317d3aa25bfff8eb203316"} Jul 16 04:41:24.890125 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:24.889696 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nsrxl" event={"ID":"42f5e13d-20f3-46ed-ab64-13ce54e5e719","Type":"ContainerStarted","Data":"01acddbf6431bbcd2931bda5d3f1e02089b730455d3c78e1ca6655450214a50c"} Jul 16 04:41:24.892878 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:24.892852 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerStarted","Data":"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7"} Jul 16 04:41:24.893006 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:24.892907 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerStarted","Data":"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b"} Jul 16 04:41:24.893006 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:24.892934 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerStarted","Data":"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59"} Jul 16 04:41:24.893006 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:24.893002 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerStarted","Data":"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f"} Jul 16 04:41:24.919946 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:24.919892 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nsrxl" podStartSLOduration=130.115838068 podStartE2EDuration="2m11.919876847s" totalImagesPullingTime="1.804038779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:41:24.919649965 +0000 UTC m=+132.237698522" watchObservedRunningTime="2026-07-16 04:41:24.919876847 +0000 UTC m=+132.237925429" Jul 16 04:41:25.027627 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:25.027571 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.391567059 podStartE2EDuration="12.027554025s" totalImagesPullingTime="7.621115798s" totalInitContainerRuntime="14.871168ms" isStatefulPod=false podCreationTimestamp="2026-07-16 04:41:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:41:25.024300278 +0000 UTC m=+132.342348847" watchObservedRunningTime="2026-07-16 04:41:25.027554025 +0000 UTC m=+132.345602582" Jul 16 04:41:25.288527 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:25.288500 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:41:27.668193 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:27.668162 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b679c4ddd-d64dk_fcf7a78a-b00a-4419-923e-5cbde11f9f85/router/0.log" Jul 16 04:41:27.702029 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:41:27.702003 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qpks8_f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66/serve-healthcheck-canary/0.log" Jul 16 04:42:15.288731 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:15.288641 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:15.304394 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:15.304362 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:16.099933 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:16.099908 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:32.435628 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.435594 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 04:42:32.436256 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.435933 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="prometheus" containerID="cri-o://9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" gracePeriod=600 Jul 16 04:42:32.436256 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.435968 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy" containerID="cri-o://577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" gracePeriod=600 Jul 16 04:42:32.436256 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.435996 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="thanos-sidecar" containerID="cri-o://cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" gracePeriod=600 Jul 16 04:42:32.436256 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.436068 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="config-reloader" containerID="cri-o://fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" gracePeriod=600 Jul 16 04:42:32.436256 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.436123 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" gracePeriod=600 Jul 16 04:42:32.436256 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.436105 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy-web" containerID="cri-o://41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" gracePeriod=600 Jul 16 04:42:32.696548 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.696516 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:32.830822 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.830744 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-thanos-prometheus-http-client-file\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.830822 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.830822 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-metrics-client-certs\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831073 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.830844 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-db\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831073 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.830869 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-2\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831073 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.830886 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-tls\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831073 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.830913 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831073 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.830940 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831073 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.830995 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-tls-assets\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831073 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831036 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831076 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-1\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831129 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-grpc-tls\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831169 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-metrics-client-ca\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831199 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-web-config\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831219 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831232 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-trusted-ca-bundle\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831264 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config-out\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831295 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-serving-certs-ca-bundle\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831326 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvkdc\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-kube-api-access-fvkdc\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831356 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-0\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831389 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-kube-rbac-proxy\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.831487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831417 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-kubelet-serving-ca-bundle\") pod \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\" (UID: \"3d8a61b0-b530-48fe-9bc8-c3567fe286e1\") " Jul 16 04:42:32.832231 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.831683 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.832231 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.832145 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 04:42:32.832616 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.832369 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 04:42:32.833310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.833279 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 04:42:32.833826 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.833801 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 04:42:32.833921 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.833903 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 04:42:32.834180 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.834156 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 04:42:32.834339 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.834316 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 04:42:32.834436 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.834408 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 04:42:32.834493 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.834473 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 04:42:32.835659 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.835620 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 04:42:32.835872 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.835686 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 04:42:32.835872 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.835686 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config" (OuterVolumeSpecName: "config") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 04:42:32.835872 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.835811 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 04:42:32.836036 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.835962 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 04:42:32.836036 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.835972 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-kube-api-access-fvkdc" (OuterVolumeSpecName: "kube-api-access-fvkdc") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "kube-api-access-fvkdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 04:42:32.836125 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.836044 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 04:42:32.836269 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.836252 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 04:42:32.836406 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.836391 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config-out" (OuterVolumeSpecName: "config-out") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 04:42:32.845831 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.845805 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-web-config" (OuterVolumeSpecName: "web-config") pod "3d8a61b0-b530-48fe-9bc8-c3567fe286e1" (UID: "3d8a61b0-b530-48fe-9bc8-c3567fe286e1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 04:42:32.932947 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.932911 2585 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-metrics-client-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.932947 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.932943 2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-web-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.932947 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.932953 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.932963 2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config-out\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.932973 2585 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.932983 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkdc\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-kube-api-access-fvkdc\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.932992 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933000 2585 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-kube-rbac-proxy\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933008 2585 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933017 2585 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933025 2585 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-metrics-client-certs\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933034 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-db\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933043 2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933052 2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933061 2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933070 2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-tls-assets\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933078 2585 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-config\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933087 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:32.933169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:32.933095 2585 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d8a61b0-b530-48fe-9bc8-c3567fe286e1-secret-grpc-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 04:42:33.150684 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150640 2585 generic.go:356] "Generic (PLEG): container finished" podID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerID="93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" exitCode=0 Jul 16 04:42:33.150684 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150674 2585 generic.go:356] "Generic (PLEG): container finished" podID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerID="577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" exitCode=0 Jul 16 04:42:33.150684 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150683 2585 generic.go:356] "Generic (PLEG): container finished" podID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerID="41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" exitCode=0 Jul 16 04:42:33.150684 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150692 2585 generic.go:356] "Generic (PLEG): container finished" podID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerID="cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" exitCode=0 Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150700 2585 generic.go:356] "Generic (PLEG): container finished" podID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerID="fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" exitCode=0 Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150708 2585 generic.go:356] "Generic (PLEG): container finished" podID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerID="9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" exitCode=0 Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150720 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerDied","Data":"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7"} Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150797 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150835 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerDied","Data":"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b"} Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150883 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerDied","Data":"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59"} Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150907 2585 scope.go:117] "RemoveContainer" containerID="93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150921 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerDied","Data":"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f"} Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.150966 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerDied","Data":"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd"} Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.151010 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerDied","Data":"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f"} Jul 16 04:42:33.151057 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.151051 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d8a61b0-b530-48fe-9bc8-c3567fe286e1","Type":"ContainerDied","Data":"442192912c818e3af56b10e99444ce0de5b7d959f0432747f33724fbc3448ab6"} Jul 16 04:42:33.159094 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.159070 2585 scope.go:117] "RemoveContainer" containerID="577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" Jul 16 04:42:33.167626 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.167603 2585 scope.go:117] "RemoveContainer" containerID="41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" Jul 16 04:42:33.177104 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.177080 2585 scope.go:117] "RemoveContainer" containerID="cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" Jul 16 04:42:33.184335 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.184312 2585 scope.go:117] "RemoveContainer" containerID="fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" Jul 16 04:42:33.187150 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.187103 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 04:42:33.191952 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.191931 2585 scope.go:117] "RemoveContainer" containerID="9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" Jul 16 04:42:33.197527 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.197500 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 04:42:33.200177 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.200155 2585 scope.go:117] "RemoveContainer" containerID="aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929" Jul 16 04:42:33.210433 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.210407 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" path="/var/lib/kubelet/pods/3d8a61b0-b530-48fe-9bc8-c3567fe286e1/volumes" Jul 16 04:42:33.212549 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.212530 2585 scope.go:117] "RemoveContainer" containerID="93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" Jul 16 04:42:33.212862 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:42:33.212833 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": container with ID starting with 93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7 not found: ID does not exist" containerID="93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" Jul 16 04:42:33.212923 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.212873 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7"} err="failed to get container status \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": rpc error: code = NotFound desc = could not find container \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": container with ID starting with 93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7 not found: ID does not exist" Jul 16 04:42:33.212923 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.212917 2585 scope.go:117] "RemoveContainer" containerID="577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" Jul 16 04:42:33.213221 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:42:33.213198 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": container with ID starting with 577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b not found: ID does not exist" containerID="577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" Jul 16 04:42:33.213306 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.213224 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b"} err="failed to get container status \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": rpc error: code = NotFound desc = could not find container \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": container with ID starting with 577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b not found: ID does not exist" Jul 16 04:42:33.213306 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.213240 2585 scope.go:117] "RemoveContainer" containerID="41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" Jul 16 04:42:33.213461 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:42:33.213445 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": container with ID starting with 41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59 not found: ID does not exist" containerID="41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" Jul 16 04:42:33.213521 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.213468 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59"} err="failed to get container status \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": rpc error: code = NotFound desc = could not find container \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": container with ID starting with 41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59 not found: ID does not exist" Jul 16 04:42:33.213521 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.213490 2585 scope.go:117] "RemoveContainer" containerID="cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" Jul 16 04:42:33.213718 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:42:33.213688 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": container with ID starting with cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f not found: ID does not exist" containerID="cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" Jul 16 04:42:33.213756 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.213722 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f"} err="failed to get container status \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": rpc error: code = NotFound desc = could not find container \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": container with ID starting with cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f not found: ID does not exist" Jul 16 04:42:33.213756 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.213740 2585 scope.go:117] "RemoveContainer" containerID="fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" Jul 16 04:42:33.213987 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:42:33.213970 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": container with ID starting with fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd not found: ID does not exist" containerID="fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" Jul 16 04:42:33.214038 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.213992 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd"} err="failed to get container status \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": rpc error: code = NotFound desc = could not find container \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": container with ID starting with fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd not found: ID does not exist" Jul 16 04:42:33.214038 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.214008 2585 scope.go:117] "RemoveContainer" containerID="9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" Jul 16 04:42:33.214214 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:42:33.214198 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": container with ID starting with 9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f not found: ID does not exist" containerID="9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" Jul 16 04:42:33.214279 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.214220 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f"} err="failed to get container status \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": rpc error: code = NotFound desc = could not find container \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": container with ID starting with 9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f not found: ID does not exist" Jul 16 04:42:33.214279 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.214239 2585 scope.go:117] "RemoveContainer" containerID="aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929" Jul 16 04:42:33.214459 ip-10-0-140-3 kubenswrapper[2585]: E0716 04:42:33.214445 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": container with ID starting with aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929 not found: ID does not exist" containerID="aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929" Jul 16 04:42:33.214497 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.214463 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929"} err="failed to get container status \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": rpc error: code = NotFound desc = could not find container \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": container with ID starting with aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929 not found: ID does not exist" Jul 16 04:42:33.214497 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.214478 2585 scope.go:117] "RemoveContainer" containerID="93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" Jul 16 04:42:33.214685 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.214667 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7"} err="failed to get container status \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": rpc error: code = NotFound desc = could not find container \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": container with ID starting with 93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7 not found: ID does not exist" Jul 16 04:42:33.214724 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.214686 2585 scope.go:117] "RemoveContainer" containerID="577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" Jul 16 04:42:33.214927 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.214908 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b"} err="failed to get container status \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": rpc error: code = NotFound desc = could not find container \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": container with ID starting with 577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b not found: ID does not exist" Jul 16 04:42:33.215004 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.214928 2585 scope.go:117] "RemoveContainer" containerID="41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" Jul 16 04:42:33.215134 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.215118 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59"} err="failed to get container status \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": rpc error: code = NotFound desc = could not find container \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": container with ID starting with 41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59 not found: ID does not exist" Jul 16 04:42:33.215182 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.215135 2585 scope.go:117] "RemoveContainer" containerID="cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" Jul 16 04:42:33.215348 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.215330 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f"} err="failed to get container status \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": rpc error: code = NotFound desc = could not find container \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": container with ID starting with cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f not found: ID does not exist" Jul 16 04:42:33.215390 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.215349 2585 scope.go:117] "RemoveContainer" containerID="fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" Jul 16 04:42:33.215553 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.215537 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd"} err="failed to get container status \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": rpc error: code = NotFound desc = could not find container \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": container with ID starting with fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd not found: ID does not exist" Jul 16 04:42:33.215620 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.215555 2585 scope.go:117] "RemoveContainer" containerID="9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" Jul 16 04:42:33.215800 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.215758 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f"} err="failed to get container status \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": rpc error: code = NotFound desc = could not find container \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": container with ID starting with 9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f not found: ID does not exist" Jul 16 04:42:33.215879 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.215803 2585 scope.go:117] "RemoveContainer" containerID="aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929" Jul 16 04:42:33.215997 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.215981 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929"} err="failed to get container status \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": rpc error: code = NotFound desc = could not find container \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": container with ID starting with aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929 not found: ID does not exist" Jul 16 04:42:33.216062 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.216000 2585 scope.go:117] "RemoveContainer" containerID="93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" Jul 16 04:42:33.216187 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.216171 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7"} err="failed to get container status \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": rpc error: code = NotFound desc = could not find container \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": container with ID starting with 93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7 not found: ID does not exist" Jul 16 04:42:33.216250 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.216188 2585 scope.go:117] "RemoveContainer" containerID="577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" Jul 16 04:42:33.216374 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.216359 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b"} err="failed to get container status \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": rpc error: code = NotFound desc = could not find container \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": container with ID starting with 577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b not found: ID does not exist" Jul 16 04:42:33.216442 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.216377 2585 scope.go:117] "RemoveContainer" containerID="41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" Jul 16 04:42:33.216581 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.216557 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59"} err="failed to get container status \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": rpc error: code = NotFound desc = could not find container \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": container with ID starting with 41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59 not found: ID does not exist" Jul 16 04:42:33.216631 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.216583 2585 scope.go:117] "RemoveContainer" containerID="cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" Jul 16 04:42:33.216860 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.216842 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f"} err="failed to get container status \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": rpc error: code = NotFound desc = could not find container \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": container with ID starting with cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f not found: ID does not exist" Jul 16 04:42:33.216906 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.216860 2585 scope.go:117] "RemoveContainer" containerID="fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" Jul 16 04:42:33.217073 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217057 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd"} err="failed to get container status \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": rpc error: code = NotFound desc = could not find container \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": container with ID starting with fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd not found: ID does not exist" Jul 16 04:42:33.217144 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217076 2585 scope.go:117] "RemoveContainer" containerID="9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" Jul 16 04:42:33.217320 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217302 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f"} err="failed to get container status \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": rpc error: code = NotFound desc = could not find container \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": container with ID starting with 9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f not found: ID does not exist" Jul 16 04:42:33.217371 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217321 2585 scope.go:117] "RemoveContainer" containerID="aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929" Jul 16 04:42:33.217534 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217514 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929"} err="failed to get container status \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": rpc error: code = NotFound desc = could not find container \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": container with ID starting with aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929 not found: ID does not exist" Jul 16 04:42:33.217601 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217535 2585 scope.go:117] "RemoveContainer" containerID="93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" Jul 16 04:42:33.217750 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217734 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7"} err="failed to get container status \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": rpc error: code = NotFound desc = could not find container \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": container with ID starting with 93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7 not found: ID does not exist" Jul 16 04:42:33.217812 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217749 2585 scope.go:117] "RemoveContainer" containerID="577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" Jul 16 04:42:33.217984 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217967 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b"} err="failed to get container status \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": rpc error: code = NotFound desc = could not find container \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": container with ID starting with 577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b not found: ID does not exist" Jul 16 04:42:33.218044 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.217985 2585 scope.go:117] "RemoveContainer" containerID="41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" Jul 16 04:42:33.218199 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.218180 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59"} err="failed to get container status \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": rpc error: code = NotFound desc = could not find container \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": container with ID starting with 41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59 not found: ID does not exist" Jul 16 04:42:33.218249 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.218199 2585 scope.go:117] "RemoveContainer" containerID="cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" Jul 16 04:42:33.218391 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.218373 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f"} err="failed to get container status \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": rpc error: code = NotFound desc = could not find container \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": container with ID starting with cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f not found: ID does not exist" Jul 16 04:42:33.218456 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.218392 2585 scope.go:117] "RemoveContainer" containerID="fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" Jul 16 04:42:33.218616 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.218599 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd"} err="failed to get container status \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": rpc error: code = NotFound desc = could not find container \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": container with ID starting with fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd not found: ID does not exist" Jul 16 04:42:33.218661 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.218617 2585 scope.go:117] "RemoveContainer" containerID="9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" Jul 16 04:42:33.218829 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.218812 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f"} err="failed to get container status \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": rpc error: code = NotFound desc = could not find container \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": container with ID starting with 9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f not found: ID does not exist" Jul 16 04:42:33.218882 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.218828 2585 scope.go:117] "RemoveContainer" containerID="aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929" Jul 16 04:42:33.219020 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219004 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929"} err="failed to get container status \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": rpc error: code = NotFound desc = could not find container \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": container with ID starting with aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929 not found: ID does not exist" Jul 16 04:42:33.219063 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219022 2585 scope.go:117] "RemoveContainer" containerID="93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" Jul 16 04:42:33.219221 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219205 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7"} err="failed to get container status \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": rpc error: code = NotFound desc = could not find container \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": container with ID starting with 93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7 not found: ID does not exist" Jul 16 04:42:33.219266 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219221 2585 scope.go:117] "RemoveContainer" containerID="577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" Jul 16 04:42:33.219392 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219376 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b"} err="failed to get container status \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": rpc error: code = NotFound desc = could not find container \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": container with ID starting with 577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b not found: ID does not exist" Jul 16 04:42:33.219392 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219390 2585 scope.go:117] "RemoveContainer" containerID="41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" Jul 16 04:42:33.219564 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219550 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59"} err="failed to get container status \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": rpc error: code = NotFound desc = could not find container \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": container with ID starting with 41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59 not found: ID does not exist" Jul 16 04:42:33.219617 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219564 2585 scope.go:117] "RemoveContainer" containerID="cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" Jul 16 04:42:33.219787 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219753 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f"} err="failed to get container status \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": rpc error: code = NotFound desc = could not find container \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": container with ID starting with cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f not found: ID does not exist" Jul 16 04:42:33.219787 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219779 2585 scope.go:117] "RemoveContainer" containerID="fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" Jul 16 04:42:33.219969 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219951 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd"} err="failed to get container status \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": rpc error: code = NotFound desc = could not find container \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": container with ID starting with fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd not found: ID does not exist" Jul 16 04:42:33.220012 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.219968 2585 scope.go:117] "RemoveContainer" containerID="9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" Jul 16 04:42:33.220211 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.220191 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f"} err="failed to get container status \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": rpc error: code = NotFound desc = could not find container \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": container with ID starting with 9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f not found: ID does not exist" Jul 16 04:42:33.220263 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.220212 2585 scope.go:117] "RemoveContainer" containerID="aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929" Jul 16 04:42:33.220415 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.220395 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929"} err="failed to get container status \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": rpc error: code = NotFound desc = could not find container \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": container with ID starting with aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929 not found: ID does not exist" Jul 16 04:42:33.220462 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.220417 2585 scope.go:117] "RemoveContainer" containerID="93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7" Jul 16 04:42:33.220598 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.220584 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7"} err="failed to get container status \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": rpc error: code = NotFound desc = could not find container \"93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7\": container with ID starting with 93deb9aba1afef595d96d7d7601ba6a730c1e0685cda9e84296631908b5736a7 not found: ID does not exist" Jul 16 04:42:33.220642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.220599 2585 scope.go:117] "RemoveContainer" containerID="577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b" Jul 16 04:42:33.220854 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.220837 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b"} err="failed to get container status \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": rpc error: code = NotFound desc = could not find container \"577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b\": container with ID starting with 577e34f60ced133fcc5ffd748b6be6b797e491faae621e48240b1987ec46ac2b not found: ID does not exist" Jul 16 04:42:33.220854 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.220854 2585 scope.go:117] "RemoveContainer" containerID="41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59" Jul 16 04:42:33.221071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.221056 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59"} err="failed to get container status \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": rpc error: code = NotFound desc = could not find container \"41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59\": container with ID starting with 41e46e6f06abc65de77a0f4007103ad107614c889933d4e51b05a7c14fbf6c59 not found: ID does not exist" Jul 16 04:42:33.221121 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.221071 2585 scope.go:117] "RemoveContainer" containerID="cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f" Jul 16 04:42:33.221292 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.221275 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f"} err="failed to get container status \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": rpc error: code = NotFound desc = could not find container \"cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f\": container with ID starting with cb4bb0fe9216c965f79f69519a4908f216b9fff7f257d347fd207df7af35110f not found: ID does not exist" Jul 16 04:42:33.221334 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.221294 2585 scope.go:117] "RemoveContainer" containerID="fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd" Jul 16 04:42:33.221481 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.221466 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd"} err="failed to get container status \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": rpc error: code = NotFound desc = could not find container \"fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd\": container with ID starting with fa1e24c1bcaf9b51ff4f47bf603c5c23f061e9551990cd1d369b04891ca87bfd not found: ID does not exist" Jul 16 04:42:33.221523 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.221482 2585 scope.go:117] "RemoveContainer" containerID="9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f" Jul 16 04:42:33.221668 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.221651 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f"} err="failed to get container status \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": rpc error: code = NotFound desc = could not find container \"9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f\": container with ID starting with 9e55c3bb3ec1fde5ad5947dd109e26ac4b13b1f41254b187a71920ce16dba24f not found: ID does not exist" Jul 16 04:42:33.221716 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.221668 2585 scope.go:117] "RemoveContainer" containerID="aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929" Jul 16 04:42:33.221895 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.221860 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929"} err="failed to get container status \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": rpc error: code = NotFound desc = could not find container \"aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929\": container with ID starting with aecda11d4ebf696aab9d43916a1aca758a7058ed96de9157898d5483f999f929 not found: ID does not exist" Jul 16 04:42:33.247572 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247547 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 04:42:33.247860 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247845 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="init-config-reloader" Jul 16 04:42:33.247935 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247861 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="init-config-reloader" Jul 16 04:42:33.247935 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247875 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="config-reloader" Jul 16 04:42:33.247935 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247884 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="config-reloader" Jul 16 04:42:33.247935 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247927 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="prometheus" Jul 16 04:42:33.247935 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247933 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="prometheus" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247939 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy-web" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247945 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy-web" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247961 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247970 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247980 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="thanos-sidecar" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247986 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="thanos-sidecar" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247994 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy-thanos" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.247999 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy-thanos" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.248053 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="config-reloader" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.248062 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="prometheus" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.248071 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="thanos-sidecar" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.248077 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy-web" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.248083 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy-thanos" Jul 16 04:42:33.248153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.248089 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8a61b0-b530-48fe-9bc8-c3567fe286e1" containerName="kube-rbac-proxy" Jul 16 04:42:33.252615 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.252592 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.264489 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.264467 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 04:42:33.264690 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.264521 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 04:42:33.264690 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.264627 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 04:42:33.264837 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.264734 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fb5tv7d79usbk\"" Jul 16 04:42:33.264896 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.264855 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 04:42:33.264952 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.264932 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 04:42:33.265414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.265364 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 04:42:33.265414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.265371 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 04:42:33.265414 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.265396 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 04:42:33.265682 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.265397 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 04:42:33.265831 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.265697 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 04:42:33.265903 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.265837 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 04:42:33.265962 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.265899 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 04:42:33.266010 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.265959 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8pkrn\"" Jul 16 04:42:33.266585 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.266569 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 04:42:33.272522 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.272495 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 04:42:33.275123 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.275103 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 04:42:33.285047 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.285022 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 04:42:33.336279 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336245 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/212248e0-9ac7-4a1d-b190-9eebc8d3d481-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336279 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336285 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336544 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336308 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-web-config\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336544 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336361 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336544 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336442 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336544 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336470 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336544 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336544 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336535 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336560 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336585 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336632 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336661 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/212248e0-9ac7-4a1d-b190-9eebc8d3d481-config-out\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336694 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336748 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-config\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.336808 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336802 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng58t\" (UniqueName: \"kubernetes.io/projected/212248e0-9ac7-4a1d-b190-9eebc8d3d481-kube-api-access-ng58t\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.337113 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336872 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.337113 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336910 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.337113 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.336937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.438395 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.438395 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438338 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.438395 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438355 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/212248e0-9ac7-4a1d-b190-9eebc8d3d481-config-out\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.438395 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438372 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.438395 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438398 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438415 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-config\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438430 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438447 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng58t\" (UniqueName: \"kubernetes.io/projected/212248e0-9ac7-4a1d-b190-9eebc8d3d481-kube-api-access-ng58t\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438475 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438504 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438528 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/212248e0-9ac7-4a1d-b190-9eebc8d3d481-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-web-config\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438638 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438720 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438759 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439088 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438823 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439800 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.438849 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439800 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.439034 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439800 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.439563 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.439800 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.439681 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.440880 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.440541 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.440880 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.440573 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.440880 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.440837 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.441295 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.441272 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.441938 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.441753 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-config\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.441938 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.441760 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.441938 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.441893 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.442274 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.442250 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.443827 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.443807 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/212248e0-9ac7-4a1d-b190-9eebc8d3d481-config-out\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.444103 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.444080 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/212248e0-9ac7-4a1d-b190-9eebc8d3d481-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.444204 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.444170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.444274 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.444254 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.444324 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.444307 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.444359 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.444336 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-web-config\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.445075 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.445054 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/212248e0-9ac7-4a1d-b190-9eebc8d3d481-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.445559 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.445544 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/212248e0-9ac7-4a1d-b190-9eebc8d3d481-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.452896 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.452873 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng58t\" (UniqueName: \"kubernetes.io/projected/212248e0-9ac7-4a1d-b190-9eebc8d3d481-kube-api-access-ng58t\") pod \"prometheus-k8s-0\" (UID: \"212248e0-9ac7-4a1d-b190-9eebc8d3d481\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.562849 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.562811 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:42:33.724513 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:33.724484 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 04:42:33.725383 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:42:33.725349 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212248e0_9ac7_4a1d_b190_9eebc8d3d481.slice/crio-f5890e237f3e8d90cdf38c63d90bab7d3dd407e8f7381966f29a0b5df4d62bf6 WatchSource:0}: Error finding container f5890e237f3e8d90cdf38c63d90bab7d3dd407e8f7381966f29a0b5df4d62bf6: Status 404 returned error can't find the container with id f5890e237f3e8d90cdf38c63d90bab7d3dd407e8f7381966f29a0b5df4d62bf6 Jul 16 04:42:34.155854 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:34.155814 2585 generic.go:356] "Generic (PLEG): container finished" podID="212248e0-9ac7-4a1d-b190-9eebc8d3d481" containerID="4cde90f28bcaa606ef85f50743b3cbf5c198729ce1b642aa92bc9bcffe5b621b" exitCode=0 Jul 16 04:42:34.156031 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:34.155913 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"212248e0-9ac7-4a1d-b190-9eebc8d3d481","Type":"ContainerDied","Data":"4cde90f28bcaa606ef85f50743b3cbf5c198729ce1b642aa92bc9bcffe5b621b"} Jul 16 04:42:34.156031 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:34.156000 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"212248e0-9ac7-4a1d-b190-9eebc8d3d481","Type":"ContainerStarted","Data":"f5890e237f3e8d90cdf38c63d90bab7d3dd407e8f7381966f29a0b5df4d62bf6"} Jul 16 04:42:35.164609 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:35.164576 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"212248e0-9ac7-4a1d-b190-9eebc8d3d481","Type":"ContainerStarted","Data":"c302bceff69449d6e34050d7e76ebc266b21bcc8d957ae34dbf61d8aa07bbb56"} Jul 16 04:42:35.165009 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:35.164664 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"212248e0-9ac7-4a1d-b190-9eebc8d3d481","Type":"ContainerStarted","Data":"85a08c0b8c762ccadfa9c90eb2e5245e39488d331517414e10653a207a8bb1db"} Jul 16 04:42:35.165009 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:35.164692 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"212248e0-9ac7-4a1d-b190-9eebc8d3d481","Type":"ContainerStarted","Data":"f7c21e6d2a76ec276263522bf7535f3a1dd92f6a15a10935bba6a5064fde0b63"} Jul 16 04:42:35.165009 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:35.164720 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"212248e0-9ac7-4a1d-b190-9eebc8d3d481","Type":"ContainerStarted","Data":"7e1297a52f0bbc80eb1ceabdd9576099b38072d939ce41053aa318c732a1855d"} Jul 16 04:42:35.165009 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:35.164743 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"212248e0-9ac7-4a1d-b190-9eebc8d3d481","Type":"ContainerStarted","Data":"2e5a06f2309ca86ba0c5ffdc27aca59d0f1139328c41960e86b902422ed308a1"} Jul 16 04:42:35.165009 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:35.164781 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"212248e0-9ac7-4a1d-b190-9eebc8d3d481","Type":"ContainerStarted","Data":"f6abbe974a2950c122a74330a610bbc1aadb70edb941305beed2fe6c119d27b6"} Jul 16 04:42:35.201617 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:35.201544 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.182804955 podStartE2EDuration="2.201521737s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.716782ms" isStatefulPod=false podCreationTimestamp="2026-07-16 04:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:42:35.200295392 +0000 UTC m=+202.518343982" watchObservedRunningTime="2026-07-16 04:42:35.201521737 +0000 UTC m=+202.519570298" Jul 16 04:42:38.563055 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:42:38.563019 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:43:33.563339 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:43:33.563290 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:43:33.578576 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:43:33.578548 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:43:34.393814 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:43:34.393788 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 04:44:13.099681 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:13.099652 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:44:13.106304 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:13.106281 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:44:13.108182 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:13.108166 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:44:13.108972 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:13.108943 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 04:44:13.116044 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:13.116026 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:44:31.406723 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.406688 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd"] Jul 16 04:44:31.408739 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.408722 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.413604 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.413585 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 04:44:31.413981 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.413965 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 04:44:31.414222 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.414208 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 04:44:31.414344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.414325 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x5n9f\"" Jul 16 04:44:31.414652 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.414638 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 04:44:31.432856 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.432833 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd"] Jul 16 04:44:31.471505 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.471465 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdhg\" (UniqueName: \"kubernetes.io/projected/c472e523-ede4-4668-8be0-d08e19d38054-kube-api-access-7pdhg\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.471665 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.471510 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c472e523-ede4-4668-8be0-d08e19d38054-manifests\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.471665 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.471546 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c472e523-ede4-4668-8be0-d08e19d38054-tmp\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.471665 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.471626 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c472e523-ede4-4668-8be0-d08e19d38054-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.471665 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.471644 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c472e523-ede4-4668-8be0-d08e19d38054-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.572240 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.572205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c472e523-ede4-4668-8be0-d08e19d38054-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.572405 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.572248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c472e523-ede4-4668-8be0-d08e19d38054-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.572405 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.572278 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdhg\" (UniqueName: \"kubernetes.io/projected/c472e523-ede4-4668-8be0-d08e19d38054-kube-api-access-7pdhg\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.572405 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.572298 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c472e523-ede4-4668-8be0-d08e19d38054-manifests\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.572552 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.572440 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c472e523-ede4-4668-8be0-d08e19d38054-tmp\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.572715 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.572689 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c472e523-ede4-4668-8be0-d08e19d38054-manifests\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.574612 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.574587 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c472e523-ede4-4668-8be0-d08e19d38054-tmp\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.574719 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.574682 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c472e523-ede4-4668-8be0-d08e19d38054-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.574866 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.574844 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c472e523-ede4-4668-8be0-d08e19d38054-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.582797 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.582754 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdhg\" (UniqueName: \"kubernetes.io/projected/c472e523-ede4-4668-8be0-d08e19d38054-kube-api-access-7pdhg\") pod \"opendatahub-operator-controller-manager-7b4b7cb559-kc8fd\" (UID: \"c472e523-ede4-4668-8be0-d08e19d38054\") " pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.718359 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.718279 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:31.864751 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.864714 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd"] Jul 16 04:44:31.868698 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:44:31.868668 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc472e523_ede4_4668_8be0_d08e19d38054.slice/crio-36ab4e9536321e8cb852d128eeb5ea04a8d40e7db261bd7cbca8a87f06bdefb4 WatchSource:0}: Error finding container 36ab4e9536321e8cb852d128eeb5ea04a8d40e7db261bd7cbca8a87f06bdefb4: Status 404 returned error can't find the container with id 36ab4e9536321e8cb852d128eeb5ea04a8d40e7db261bd7cbca8a87f06bdefb4 Jul 16 04:44:31.870606 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:31.870589 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 04:44:32.591696 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:32.591642 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" event={"ID":"c472e523-ede4-4668-8be0-d08e19d38054","Type":"ContainerStarted","Data":"36ab4e9536321e8cb852d128eeb5ea04a8d40e7db261bd7cbca8a87f06bdefb4"} Jul 16 04:44:35.609325 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:35.609280 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" event={"ID":"c472e523-ede4-4668-8be0-d08e19d38054","Type":"ContainerStarted","Data":"d9d1275b983c3014e296f1673014de657a224fadc5354a68c5f773f81dc6cfef"} Jul 16 04:44:36.614950 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:36.614913 2585 generic.go:356] "Generic (PLEG): container finished" podID="c472e523-ede4-4668-8be0-d08e19d38054" containerID="d9d1275b983c3014e296f1673014de657a224fadc5354a68c5f773f81dc6cfef" exitCode=0 Jul 16 04:44:36.615425 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:36.614976 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" event={"ID":"c472e523-ede4-4668-8be0-d08e19d38054","Type":"ContainerDied","Data":"d9d1275b983c3014e296f1673014de657a224fadc5354a68c5f773f81dc6cfef"} Jul 16 04:44:37.620461 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:37.620425 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" event={"ID":"c472e523-ede4-4668-8be0-d08e19d38054","Type":"ContainerStarted","Data":"da15a66414f7158e8a74f40c2cac79df6b7f14d96b56ae8121cf7fcc51679774"} Jul 16 04:44:37.620868 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:37.620518 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:37.648906 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:37.648841 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" podStartSLOduration=2.370237699 podStartE2EDuration="6.648821506s" totalImagesPullingTime="3.272356503s" totalInitContainerRuntime="1.006227304s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:44:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:44:37.648322961 +0000 UTC m=+324.966371573" watchObservedRunningTime="2026-07-16 04:44:37.648821506 +0000 UTC m=+324.966870065" Jul 16 04:44:48.627063 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:48.627032 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b4b7cb559-kc8fd" Jul 16 04:44:59.470812 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.470762 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc"] Jul 16 04:44:59.480091 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.480070 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.484646 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.484430 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 04:44:59.484646 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.484519 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 04:44:59.484646 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.484553 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 04:44:59.484969 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.484753 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 04:44:59.485083 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.485063 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 04:44:59.485218 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.485196 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vhszm\"" Jul 16 04:44:59.486678 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.486656 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc"] Jul 16 04:44:59.606055 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.606024 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2090408-5fb5-433d-acdf-dd062fb418a7-metrics-cert\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.606228 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.606079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4v92\" (UniqueName: \"kubernetes.io/projected/b2090408-5fb5-433d-acdf-dd062fb418a7-kube-api-access-t4v92\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.606228 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.606157 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2090408-5fb5-433d-acdf-dd062fb418a7-cert\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.606228 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.606198 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b2090408-5fb5-433d-acdf-dd062fb418a7-manager-config\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.706897 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.706869 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2090408-5fb5-433d-acdf-dd062fb418a7-cert\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.707071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.706907 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b2090408-5fb5-433d-acdf-dd062fb418a7-manager-config\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.707071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.706937 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2090408-5fb5-433d-acdf-dd062fb418a7-metrics-cert\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.707071 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.706978 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v92\" (UniqueName: \"kubernetes.io/projected/b2090408-5fb5-433d-acdf-dd062fb418a7-kube-api-access-t4v92\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.707633 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.707610 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b2090408-5fb5-433d-acdf-dd062fb418a7-manager-config\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.709423 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.709397 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2090408-5fb5-433d-acdf-dd062fb418a7-cert\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.709511 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.709456 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2090408-5fb5-433d-acdf-dd062fb418a7-metrics-cert\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.725674 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.723499 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4v92\" (UniqueName: \"kubernetes.io/projected/b2090408-5fb5-433d-acdf-dd062fb418a7-kube-api-access-t4v92\") pod \"lws-controller-manager-546f8cbbbf-vqgtc\" (UID: \"b2090408-5fb5-433d-acdf-dd062fb418a7\") " pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.790953 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.790918 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:44:59.916954 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:44:59.916917 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc"] Jul 16 04:44:59.919885 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:44:59.919862 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2090408_5fb5_433d_acdf_dd062fb418a7.slice/crio-eb5d4590ec5ec4c9b78ac2bf6fa0882905bc026fffc574cbb376d3f651f7a2f5 WatchSource:0}: Error finding container eb5d4590ec5ec4c9b78ac2bf6fa0882905bc026fffc574cbb376d3f651f7a2f5: Status 404 returned error can't find the container with id eb5d4590ec5ec4c9b78ac2bf6fa0882905bc026fffc574cbb376d3f651f7a2f5 Jul 16 04:45:00.713255 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.713223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" event={"ID":"b2090408-5fb5-433d-acdf-dd062fb418a7","Type":"ContainerStarted","Data":"eb5d4590ec5ec4c9b78ac2bf6fa0882905bc026fffc574cbb376d3f651f7a2f5"} Jul 16 04:45:00.746709 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.746673 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28"] Jul 16 04:45:00.749513 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.749491 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.752144 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.752081 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 16 04:45:00.752536 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.752513 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 16 04:45:00.752999 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.752971 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 16 04:45:00.753184 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.753168 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 04:45:00.753427 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.753410 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8w6rd\"" Jul 16 04:45:00.766146 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.766119 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28"] Jul 16 04:45:00.816710 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.816678 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ctjk\" (UniqueName: \"kubernetes.io/projected/f97db3db-1a6f-40e2-b957-2041ece56b7a-kube-api-access-6ctjk\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.816710 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.816725 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.816980 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.816755 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.816980 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.816878 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97db3db-1a6f-40e2-b957-2041ece56b7a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.816980 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.816928 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.817153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.817054 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.817153 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.817100 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f97db3db-1a6f-40e2-b957-2041ece56b7a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.918220 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.918181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f97db3db-1a6f-40e2-b957-2041ece56b7a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.918387 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.918245 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ctjk\" (UniqueName: \"kubernetes.io/projected/f97db3db-1a6f-40e2-b957-2041ece56b7a-kube-api-access-6ctjk\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.918387 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.918281 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.918387 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.918322 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.918387 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.918363 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97db3db-1a6f-40e2-b957-2041ece56b7a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.918619 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.918403 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.918619 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.918461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.919169 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.919138 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.921472 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.921422 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97db3db-1a6f-40e2-b957-2041ece56b7a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.922291 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.921934 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.922291 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.922259 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.922457 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.922362 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f97db3db-1a6f-40e2-b957-2041ece56b7a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.927728 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.927677 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ctjk\" (UniqueName: \"kubernetes.io/projected/f97db3db-1a6f-40e2-b957-2041ece56b7a-kube-api-access-6ctjk\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:00.928537 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:00.928514 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97db3db-1a6f-40e2-b957-2041ece56b7a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-fwk28\" (UID: \"f97db3db-1a6f-40e2-b957-2041ece56b7a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:01.063706 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:01.063660 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:01.285064 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:45:01.284916 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97db3db_1a6f_40e2_b957_2041ece56b7a.slice/crio-7fe66b0d6c9d442826d0b461aad8e0c3ece7d71135c26dceee7259282376d9a7 WatchSource:0}: Error finding container 7fe66b0d6c9d442826d0b461aad8e0c3ece7d71135c26dceee7259282376d9a7: Status 404 returned error can't find the container with id 7fe66b0d6c9d442826d0b461aad8e0c3ece7d71135c26dceee7259282376d9a7 Jul 16 04:45:01.289292 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:01.289261 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28"] Jul 16 04:45:01.719785 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:01.719729 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" event={"ID":"f97db3db-1a6f-40e2-b957-2041ece56b7a","Type":"ContainerStarted","Data":"7fe66b0d6c9d442826d0b461aad8e0c3ece7d71135c26dceee7259282376d9a7"} Jul 16 04:45:02.726395 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:02.726297 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" event={"ID":"b2090408-5fb5-433d-acdf-dd062fb418a7","Type":"ContainerStarted","Data":"439015f96da26163d585704a74ddeafa59938edfb61dc02b502208cff084bab5"} Jul 16 04:45:02.726907 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:02.726558 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:45:02.777410 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:02.777029 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" podStartSLOduration=1.263189037 podStartE2EDuration="3.777004562s" totalImagesPullingTime="2.513815525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:45:02.776193498 +0000 UTC m=+350.094242084" watchObservedRunningTime="2026-07-16 04:45:02.777004562 +0000 UTC m=+350.095053116" Jul 16 04:45:04.370013 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:04.369963 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 04:45:04.370395 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:04.370052 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 04:45:04.745377 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:04.745291 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" event={"ID":"f97db3db-1a6f-40e2-b957-2041ece56b7a","Type":"ContainerStarted","Data":"e0fa3eb14f868e5e15171eee82685f32dbac2187886e95b1698fa48ff8cc4d5d"} Jul 16 04:45:04.745547 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:04.745462 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:04.801232 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:04.801171 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" podStartSLOduration=1.71904207 podStartE2EDuration="4.801149715s" totalImagesPullingTime="3.082107645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:45:04.798012925 +0000 UTC m=+352.116061484" watchObservedRunningTime="2026-07-16 04:45:04.801149715 +0000 UTC m=+352.119198276" Jul 16 04:45:05.751802 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:05.751702 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-fwk28" Jul 16 04:45:13.733590 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:45:13.733558 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-546f8cbbbf-vqgtc" Jul 16 04:47:22.692134 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.692100 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:47:22.695521 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.695501 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:47:22.697927 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.697903 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 04:47:22.698039 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.697998 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 04:47:22.698039 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.698009 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8r5pn\"" Jul 16 04:47:22.698794 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.698778 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 04:47:22.703561 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.703538 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:47:22.729543 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.729515 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:47:22.803924 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.803888 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1417baf-fcbb-4d81-9b92-53fd1b99417a-config-file\") pod \"limitador-limitador-54d6d988d5-bfh4b\" (UID: \"b1417baf-fcbb-4d81-9b92-53fd1b99417a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:47:22.804082 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.803932 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rrs\" (UniqueName: \"kubernetes.io/projected/b1417baf-fcbb-4d81-9b92-53fd1b99417a-kube-api-access-j6rrs\") pod \"limitador-limitador-54d6d988d5-bfh4b\" (UID: \"b1417baf-fcbb-4d81-9b92-53fd1b99417a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:47:22.904640 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.904611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rrs\" (UniqueName: \"kubernetes.io/projected/b1417baf-fcbb-4d81-9b92-53fd1b99417a-kube-api-access-j6rrs\") pod \"limitador-limitador-54d6d988d5-bfh4b\" (UID: \"b1417baf-fcbb-4d81-9b92-53fd1b99417a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:47:22.904835 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.904702 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1417baf-fcbb-4d81-9b92-53fd1b99417a-config-file\") pod \"limitador-limitador-54d6d988d5-bfh4b\" (UID: \"b1417baf-fcbb-4d81-9b92-53fd1b99417a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:47:22.905295 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.905275 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1417baf-fcbb-4d81-9b92-53fd1b99417a-config-file\") pod \"limitador-limitador-54d6d988d5-bfh4b\" (UID: \"b1417baf-fcbb-4d81-9b92-53fd1b99417a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:47:22.913907 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:22.913886 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rrs\" (UniqueName: \"kubernetes.io/projected/b1417baf-fcbb-4d81-9b92-53fd1b99417a-kube-api-access-j6rrs\") pod \"limitador-limitador-54d6d988d5-bfh4b\" (UID: \"b1417baf-fcbb-4d81-9b92-53fd1b99417a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:47:23.005642 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:23.005559 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:47:23.134732 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:23.134636 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:47:23.137485 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:47:23.137399 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1417baf_fcbb_4d81_9b92_53fd1b99417a.slice/crio-bc35c94262e706a5467e8c29e8c9708df269ee71b73cf3840b9bb774329ac813 WatchSource:0}: Error finding container bc35c94262e706a5467e8c29e8c9708df269ee71b73cf3840b9bb774329ac813: Status 404 returned error can't find the container with id bc35c94262e706a5467e8c29e8c9708df269ee71b73cf3840b9bb774329ac813 Jul 16 04:47:23.318437 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:23.318399 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" event={"ID":"b1417baf-fcbb-4d81-9b92-53fd1b99417a","Type":"ContainerStarted","Data":"bc35c94262e706a5467e8c29e8c9708df269ee71b73cf3840b9bb774329ac813"} Jul 16 04:47:26.333206 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:26.333170 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" event={"ID":"b1417baf-fcbb-4d81-9b92-53fd1b99417a","Type":"ContainerStarted","Data":"192cf1c526611eca89287944c7070b0b34402f70d30c7ec8f933b0ace49b3c27"} Jul 16 04:47:26.333704 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:26.333298 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:47:26.353819 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:26.353727 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" podStartSLOduration=1.820212892 podStartE2EDuration="4.353712519s" totalImagesPullingTime="2.533499627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:47:26.352970248 +0000 UTC m=+493.671018818" watchObservedRunningTime="2026-07-16 04:47:26.353712519 +0000 UTC m=+493.671761078" Jul 16 04:47:37.338468 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:47:37.338434 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-bfh4b" Jul 16 04:48:38.848300 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:48:38.848212 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:48:58.575089 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:48:58.575057 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:49:13.126296 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:13.126269 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:49:13.133245 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:13.133224 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:49:13.133385 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:13.133366 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:49:13.140036 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:13.140020 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:49:14.686344 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:14.686314 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:49:24.899838 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.899805 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp"] Jul 16 04:49:24.903030 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.903013 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:24.906282 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.906257 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 04:49:24.906282 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.906269 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 16 04:49:24.906282 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.906262 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q2d59\"" Jul 16 04:49:24.906487 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.906285 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 04:49:24.916230 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.916211 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp"] Jul 16 04:49:24.924663 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.924637 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:24.924745 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.924677 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hn8\" (UniqueName: \"kubernetes.io/projected/e9223697-3fd0-4e67-851e-a7b1ef10d708-kube-api-access-l6hn8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:24.924804 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.924783 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9223697-3fd0-4e67-851e-a7b1ef10d708-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:24.924844 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.924828 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:24.924887 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.924873 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:24.924974 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:24.924943 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.025824 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.025785 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9223697-3fd0-4e67-851e-a7b1ef10d708-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.026023 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.025832 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.026023 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.025867 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.026023 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.025902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.026023 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.025932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.026023 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.025948 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hn8\" (UniqueName: \"kubernetes.io/projected/e9223697-3fd0-4e67-851e-a7b1ef10d708-kube-api-access-l6hn8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.026314 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.026293 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.026412 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.026391 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.026465 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.026410 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.028113 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.028096 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9223697-3fd0-4e67-851e-a7b1ef10d708-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.028506 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.028486 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9223697-3fd0-4e67-851e-a7b1ef10d708-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.041000 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.040969 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hn8\" (UniqueName: \"kubernetes.io/projected/e9223697-3fd0-4e67-851e-a7b1ef10d708-kube-api-access-l6hn8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-74mmp\" (UID: \"e9223697-3fd0-4e67-851e-a7b1ef10d708\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.213513 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.213424 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:25.342756 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.342731 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp"] Jul 16 04:49:25.345359 ip-10-0-140-3 kubenswrapper[2585]: W0716 04:49:25.345332 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9223697_3fd0_4e67_851e_a7b1ef10d708.slice/crio-553cc7c830063ea7452aeb82d182bf362ca858d5eae7da2c6f4bcc5345817baf WatchSource:0}: Error finding container 553cc7c830063ea7452aeb82d182bf362ca858d5eae7da2c6f4bcc5345817baf: Status 404 returned error can't find the container with id 553cc7c830063ea7452aeb82d182bf362ca858d5eae7da2c6f4bcc5345817baf Jul 16 04:49:25.836125 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:25.836076 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" event={"ID":"e9223697-3fd0-4e67-851e-a7b1ef10d708","Type":"ContainerStarted","Data":"553cc7c830063ea7452aeb82d182bf362ca858d5eae7da2c6f4bcc5345817baf"} Jul 16 04:49:26.174486 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:26.174398 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:49:26.841974 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:26.841938 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" event={"ID":"e9223697-3fd0-4e67-851e-a7b1ef10d708","Type":"ContainerStarted","Data":"b918553254f00975f93d540fe40b097ce283b50b64912cea1bd567d67dd087ca"} Jul 16 04:49:26.865078 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:26.865031 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" podStartSLOduration=1.8885078069999999 podStartE2EDuration="2.865015792s" totalImagesPullingTime="976.507985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 04:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 04:49:26.863589426 +0000 UTC m=+614.181638016" watchObservedRunningTime="2026-07-16 04:49:26.865015792 +0000 UTC m=+614.183064349" Jul 16 04:49:30.482501 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:30.482469 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:49:35.214310 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:35.214279 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:35.216891 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:35.214333 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:35.226634 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:35.226615 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:35.890727 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:35.890701 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-74mmp" Jul 16 04:49:45.684853 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:45.684785 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:49:55.681242 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:49:55.681204 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:51:32.794268 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:51:32.794193 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:51:41.776522 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:51:41.776485 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:51:53.492233 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:51:53.487938 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:52:02.677468 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:52:02.677431 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:52:13.276833 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:52:13.276802 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:52:43.478615 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:52:43.478579 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:52:58.868937 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:52:58.868894 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:53:31.678649 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:53:31.678611 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:53:41.275741 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:53:41.275708 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:53:51.376394 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:53:51.376360 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:53:57.184199 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:53:57.184160 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:54:08.390241 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:54:08.390151 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:54:13.152447 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:54:13.152418 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:54:13.163984 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:54:13.163959 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:54:13.167605 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:54:13.167580 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:54:13.170518 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:54:13.170501 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:54:18.278577 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:54:18.278540 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:54:29.384606 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:54:29.384575 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:54:39.470952 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:54:39.470914 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:55:04.406436 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:55:04.406407 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:55:14.073320 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:55:14.073287 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:55:37.880664 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:55:37.880559 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:55:48.075197 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:55:48.075156 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:56:02.888881 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:56:02.888841 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:56:13.778522 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:56:13.778483 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:56:23.296964 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:56:23.296929 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:56:32.396785 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:56:32.396731 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:56:41.982592 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:56:41.982559 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:58:31.984017 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:58:31.983982 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:58:45.179283 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:58:45.179242 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:58:53.873962 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:58:53.873931 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:59:04.193311 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:04.193273 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:59:13.188913 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:13.188887 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:59:13.189366 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:13.188946 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 04:59:13.195094 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:13.195065 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:59:13.195229 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:13.195188 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 04:59:14.582400 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:14.582367 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:59:25.180425 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:25.180390 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:59:34.495046 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:34.491419 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:59:45.171938 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:45.171899 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 04:59:55.583377 ip-10-0-140-3 kubenswrapper[2585]: I0716 04:59:55.583345 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:00:00.147782 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:00.147736 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k"] Jul 16 05:00:00.151035 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:00.151011 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k" Jul 16 05:00:00.153264 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:00.153243 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7xp78\"" Jul 16 05:00:00.160509 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:00.160492 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k" Jul 16 05:00:00.167308 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:00.167282 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k"] Jul 16 05:00:00.292272 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:00.292243 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k"] Jul 16 05:00:00.294744 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:00:00.294713 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe06419_9d5c_4e34_a528_6116a85e9c46.slice/crio-f09ac9903bbd8185eb84c766fb42b9d352ac719fea6a5333c28d8b66b5faae29 WatchSource:0}: Error finding container f09ac9903bbd8185eb84c766fb42b9d352ac719fea6a5333c28d8b66b5faae29: Status 404 returned error can't find the container with id f09ac9903bbd8185eb84c766fb42b9d352ac719fea6a5333c28d8b66b5faae29 Jul 16 05:00:00.296745 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:00.296725 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 05:00:00.506456 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:00.506378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k" event={"ID":"dfe06419-9d5c-4e34-a528-6116a85e9c46","Type":"ContainerStarted","Data":"f09ac9903bbd8185eb84c766fb42b9d352ac719fea6a5333c28d8b66b5faae29"} Jul 16 05:00:05.584952 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:05.584872 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:00:12.561898 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:12.561863 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k" event={"ID":"dfe06419-9d5c-4e34-a528-6116a85e9c46","Type":"ContainerStarted","Data":"6d047706fdc5565fcefb61cec04df5807abacb5c6f17f188c685bc8fdd15e2b4"} Jul 16 05:00:12.578364 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:12.578309 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k" podStartSLOduration=0.78610385 podStartE2EDuration="12.57829389s" totalImagesPullingTime="11.79219004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:00:12.576484356 +0000 UTC m=+1259.894532937" watchObservedRunningTime="2026-07-16 05:00:12.57829389 +0000 UTC m=+1259.896342449" Jul 16 05:00:13.568065 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:13.568028 2585 generic.go:356] "Generic (PLEG): container finished" podID="dfe06419-9d5c-4e34-a528-6116a85e9c46" containerID="6d047706fdc5565fcefb61cec04df5807abacb5c6f17f188c685bc8fdd15e2b4" exitCode=0 Jul 16 05:00:13.568431 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:13.568077 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k" event={"ID":"dfe06419-9d5c-4e34-a528-6116a85e9c46","Type":"ContainerDied","Data":"6d047706fdc5565fcefb61cec04df5807abacb5c6f17f188c685bc8fdd15e2b4"} Jul 16 05:00:14.703192 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:14.703162 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k" Jul 16 05:00:15.577810 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:15.577781 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k" Jul 16 05:00:15.577976 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:15.577808 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736300-pbv9k" event={"ID":"dfe06419-9d5c-4e34-a528-6116a85e9c46","Type":"ContainerDied","Data":"f09ac9903bbd8185eb84c766fb42b9d352ac719fea6a5333c28d8b66b5faae29"} Jul 16 05:00:15.577976 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:15.577858 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f09ac9903bbd8185eb84c766fb42b9d352ac719fea6a5333c28d8b66b5faae29" Jul 16 05:00:17.980460 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:17.980424 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:00:39.883224 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:39.883192 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:00:48.585199 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:48.585168 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:00:58.076483 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:00:58.076438 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:01:10.386123 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:01:10.386086 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:01:27.978263 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:01:27.978225 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:01:37.480943 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:01:37.480857 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:01:46.781662 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:01:46.781623 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:01:57.575924 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:01:57.575887 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:02:06.476982 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:02:06.476944 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:02:15.681049 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:02:15.681001 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:02:28.985004 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:02:28.984969 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:02:38.290528 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:02:38.290490 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:02:47.478620 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:02:47.478584 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:02:58.184229 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:02:58.184192 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:03:10.180423 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:03:10.180339 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:03:17.079101 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:03:17.079059 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:03:33.579927 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:03:33.579894 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:03:43.175890 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:03:43.175852 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:03:52.577955 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:03:52.577925 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:04:01.380569 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:01.380530 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:04:10.876781 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:10.876729 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:04:13.213002 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:13.212971 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 05:04:13.214402 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:13.214377 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 05:04:13.219982 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:13.219960 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 05:04:13.221142 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:13.221126 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 05:04:20.849051 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:20.849010 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:04:29.679543 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:29.679507 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:04:39.073001 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:39.072920 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:04:59.032284 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.032253 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx"] Jul 16 05:04:59.032670 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.032562 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe06419-9d5c-4e34-a528-6116a85e9c46" containerName="cleanup" Jul 16 05:04:59.032670 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.032574 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe06419-9d5c-4e34-a528-6116a85e9c46" containerName="cleanup" Jul 16 05:04:59.032670 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.032642 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe06419-9d5c-4e34-a528-6116a85e9c46" containerName="cleanup" Jul 16 05:04:59.035861 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.035835 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.038712 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.038685 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f6882454-serving-cert\"" Jul 16 05:04:59.039475 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.039456 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47w6t\"" Jul 16 05:04:59.039891 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.039511 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 05:04:59.039891 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.039484 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 05:04:59.047813 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.047734 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx"] Jul 16 05:04:59.111041 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.111006 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92ae801-60ee-437d-8b0e-f25f43e401be-maas-api-tls\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.111216 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.111057 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjzh\" (UniqueName: \"kubernetes.io/projected/f92ae801-60ee-437d-8b0e-f25f43e401be-kube-api-access-5qjzh\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.111216 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.111101 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92ae801-60ee-437d-8b0e-f25f43e401be-openshift-service-ca\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.212493 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.212458 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92ae801-60ee-437d-8b0e-f25f43e401be-maas-api-tls\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.212493 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.212498 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjzh\" (UniqueName: \"kubernetes.io/projected/f92ae801-60ee-437d-8b0e-f25f43e401be-kube-api-access-5qjzh\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.212700 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.212517 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92ae801-60ee-437d-8b0e-f25f43e401be-openshift-service-ca\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.212700 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:04:59.212599 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-serving-cert: secret "maas-api-e2e-migrate-f6882454-serving-cert" not found Jul 16 05:04:59.212700 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:04:59.212662 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f92ae801-60ee-437d-8b0e-f25f43e401be-maas-api-tls podName:f92ae801-60ee-437d-8b0e-f25f43e401be nodeName:}" failed. No retries permitted until 2026-07-16 05:04:59.712643438 +0000 UTC m=+1547.030691976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f92ae801-60ee-437d-8b0e-f25f43e401be-maas-api-tls") pod "maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" (UID: "f92ae801-60ee-437d-8b0e-f25f43e401be") : secret "maas-api-e2e-migrate-f6882454-serving-cert" not found Jul 16 05:04:59.213127 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.213111 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92ae801-60ee-437d-8b0e-f25f43e401be-openshift-service-ca\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.224305 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.224280 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjzh\" (UniqueName: \"kubernetes.io/projected/f92ae801-60ee-437d-8b0e-f25f43e401be-kube-api-access-5qjzh\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.716896 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.716863 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92ae801-60ee-437d-8b0e-f25f43e401be-maas-api-tls\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.719277 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.719256 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92ae801-60ee-437d-8b0e-f25f43e401be-maas-api-tls\") pod \"maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:04:59.948646 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:04:59.948605 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:05:00.096049 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:00.096002 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx"] Jul 16 05:05:00.100181 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:05:00.100151 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92ae801_60ee_437d_8b0e_f25f43e401be.slice/crio-dbbf2217eb22f290dc4a8a8457ca3edd97543df151825b16a6f6fa0491ce7289 WatchSource:0}: Error finding container dbbf2217eb22f290dc4a8a8457ca3edd97543df151825b16a6f6fa0491ce7289: Status 404 returned error can't find the container with id dbbf2217eb22f290dc4a8a8457ca3edd97543df151825b16a6f6fa0491ce7289 Jul 16 05:05:00.797826 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:00.797784 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" event={"ID":"f92ae801-60ee-437d-8b0e-f25f43e401be","Type":"ContainerStarted","Data":"dbbf2217eb22f290dc4a8a8457ca3edd97543df151825b16a6f6fa0491ce7289"} Jul 16 05:05:02.284469 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:02.284430 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:05:02.808178 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:02.808144 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" event={"ID":"f92ae801-60ee-437d-8b0e-f25f43e401be","Type":"ContainerStarted","Data":"910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985"} Jul 16 05:05:02.808323 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:02.808263 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:05:02.825667 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:02.825500 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" podStartSLOduration=1.583548021 podStartE2EDuration="3.825483552s" totalImagesPullingTime="2.241935531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:05:02.824499582 +0000 UTC m=+1550.142548140" watchObservedRunningTime="2026-07-16 05:05:02.825483552 +0000 UTC m=+1550.143532111" Jul 16 05:05:08.818881 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:08.818845 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:05:10.235994 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.235956 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx"] Jul 16 05:05:10.236467 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.236173 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" podUID="f92ae801-60ee-437d-8b0e-f25f43e401be" containerName="maas-api" containerID="cri-o://910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985" gracePeriod=30 Jul 16 05:05:10.488894 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.488842 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:05:10.622249 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.622215 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qjzh\" (UniqueName: \"kubernetes.io/projected/f92ae801-60ee-437d-8b0e-f25f43e401be-kube-api-access-5qjzh\") pod \"f92ae801-60ee-437d-8b0e-f25f43e401be\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " Jul 16 05:05:10.622418 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.622289 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92ae801-60ee-437d-8b0e-f25f43e401be-maas-api-tls\") pod \"f92ae801-60ee-437d-8b0e-f25f43e401be\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " Jul 16 05:05:10.622418 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.622340 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92ae801-60ee-437d-8b0e-f25f43e401be-openshift-service-ca\") pod \"f92ae801-60ee-437d-8b0e-f25f43e401be\" (UID: \"f92ae801-60ee-437d-8b0e-f25f43e401be\") " Jul 16 05:05:10.622716 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.622692 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92ae801-60ee-437d-8b0e-f25f43e401be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f92ae801-60ee-437d-8b0e-f25f43e401be" (UID: "f92ae801-60ee-437d-8b0e-f25f43e401be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 05:05:10.624424 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.624397 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92ae801-60ee-437d-8b0e-f25f43e401be-kube-api-access-5qjzh" (OuterVolumeSpecName: "kube-api-access-5qjzh") pod "f92ae801-60ee-437d-8b0e-f25f43e401be" (UID: "f92ae801-60ee-437d-8b0e-f25f43e401be"). InnerVolumeSpecName "kube-api-access-5qjzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 05:05:10.624424 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.624418 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f92ae801-60ee-437d-8b0e-f25f43e401be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f92ae801-60ee-437d-8b0e-f25f43e401be" (UID: "f92ae801-60ee-437d-8b0e-f25f43e401be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 05:05:10.723527 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.723485 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qjzh\" (UniqueName: \"kubernetes.io/projected/f92ae801-60ee-437d-8b0e-f25f43e401be-kube-api-access-5qjzh\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:05:10.723527 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.723526 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92ae801-60ee-437d-8b0e-f25f43e401be-maas-api-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:05:10.723719 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.723540 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92ae801-60ee-437d-8b0e-f25f43e401be-openshift-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:05:10.842832 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.842801 2585 generic.go:356] "Generic (PLEG): container finished" podID="f92ae801-60ee-437d-8b0e-f25f43e401be" containerID="910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985" exitCode=0 Jul 16 05:05:10.843001 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.842863 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" event={"ID":"f92ae801-60ee-437d-8b0e-f25f43e401be","Type":"ContainerDied","Data":"910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985"} Jul 16 05:05:10.843001 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.842893 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" Jul 16 05:05:10.843001 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.842903 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx" event={"ID":"f92ae801-60ee-437d-8b0e-f25f43e401be","Type":"ContainerDied","Data":"dbbf2217eb22f290dc4a8a8457ca3edd97543df151825b16a6f6fa0491ce7289"} Jul 16 05:05:10.843001 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.842924 2585 scope.go:117] "RemoveContainer" containerID="910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985" Jul 16 05:05:10.851184 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.851165 2585 scope.go:117] "RemoveContainer" containerID="910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985" Jul 16 05:05:10.851434 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:05:10.851414 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985\": container with ID starting with 910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985 not found: ID does not exist" containerID="910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985" Jul 16 05:05:10.851471 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.851442 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985"} err="failed to get container status \"910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985\": rpc error: code = NotFound desc = could not find container \"910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985\": container with ID starting with 910e4f98479802852d4877a15d36a965ec837b6dcec6bc874264946e3e357985 not found: ID does not exist" Jul 16 05:05:10.863706 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.863679 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx"] Jul 16 05:05:10.871948 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:10.871923 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6882454-77b865cc7-4fsxx"] Jul 16 05:05:11.209746 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:11.209650 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f92ae801-60ee-437d-8b0e-f25f43e401be" path="/var/lib/kubelet/pods/f92ae801-60ee-437d-8b0e-f25f43e401be/volumes" Jul 16 05:05:11.372997 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:11.372958 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:05:16.574276 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.574244 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g"] Jul 16 05:05:16.574690 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.574589 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92ae801-60ee-437d-8b0e-f25f43e401be" containerName="maas-api" Jul 16 05:05:16.574690 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.574602 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92ae801-60ee-437d-8b0e-f25f43e401be" containerName="maas-api" Jul 16 05:05:16.574690 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.574662 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92ae801-60ee-437d-8b0e-f25f43e401be" containerName="maas-api" Jul 16 05:05:16.580951 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.580929 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:16.583537 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.583515 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 05:05:16.583666 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.583563 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9116b405-serving-cert\"" Jul 16 05:05:16.583666 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.583577 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 05:05:16.583666 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.583517 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47w6t\"" Jul 16 05:05:16.587424 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.587400 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g"] Jul 16 05:05:16.675925 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.675841 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011a33f2-468f-4359-9506-c70df101d2a4-maas-api-tls\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:16.675925 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.675911 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtddf\" (UniqueName: \"kubernetes.io/projected/011a33f2-468f-4359-9506-c70df101d2a4-kube-api-access-mtddf\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:16.676118 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.675953 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011a33f2-468f-4359-9506-c70df101d2a4-openshift-service-ca\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:16.776889 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.776854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011a33f2-468f-4359-9506-c70df101d2a4-maas-api-tls\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:16.777047 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.776915 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtddf\" (UniqueName: \"kubernetes.io/projected/011a33f2-468f-4359-9506-c70df101d2a4-kube-api-access-mtddf\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:16.777047 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.776948 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011a33f2-468f-4359-9506-c70df101d2a4-openshift-service-ca\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:16.777047 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:05:16.777004 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-serving-cert: secret "maas-api-e2e-derive-9116b405-serving-cert" not found Jul 16 05:05:16.777147 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:05:16.777072 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011a33f2-468f-4359-9506-c70df101d2a4-maas-api-tls podName:011a33f2-468f-4359-9506-c70df101d2a4 nodeName:}" failed. No retries permitted until 2026-07-16 05:05:17.277054749 +0000 UTC m=+1564.595103295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/011a33f2-468f-4359-9506-c70df101d2a4-maas-api-tls") pod "maas-api-e2e-derive-9116b405-884b69d6d-nz27g" (UID: "011a33f2-468f-4359-9506-c70df101d2a4") : secret "maas-api-e2e-derive-9116b405-serving-cert" not found Jul 16 05:05:16.777591 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.777574 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011a33f2-468f-4359-9506-c70df101d2a4-openshift-service-ca\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:16.792324 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:16.792293 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtddf\" (UniqueName: \"kubernetes.io/projected/011a33f2-468f-4359-9506-c70df101d2a4-kube-api-access-mtddf\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:17.281391 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:17.281358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011a33f2-468f-4359-9506-c70df101d2a4-maas-api-tls\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:17.284029 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:17.283993 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011a33f2-468f-4359-9506-c70df101d2a4-maas-api-tls\") pod \"maas-api-e2e-derive-9116b405-884b69d6d-nz27g\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:17.493058 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:17.493022 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:17.628223 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:17.628196 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g"] Jul 16 05:05:17.630436 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:05:17.630406 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011a33f2_468f_4359_9506_c70df101d2a4.slice/crio-a0fcd1bb5eeccb3059338aab117314cacd68355bd74581773b65c9a3b020b2eb WatchSource:0}: Error finding container a0fcd1bb5eeccb3059338aab117314cacd68355bd74581773b65c9a3b020b2eb: Status 404 returned error can't find the container with id a0fcd1bb5eeccb3059338aab117314cacd68355bd74581773b65c9a3b020b2eb Jul 16 05:05:17.631850 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:17.631829 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 05:05:17.872797 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:17.872689 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" event={"ID":"011a33f2-468f-4359-9506-c70df101d2a4","Type":"ContainerStarted","Data":"a0fcd1bb5eeccb3059338aab117314cacd68355bd74581773b65c9a3b020b2eb"} Jul 16 05:05:18.878728 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:18.878696 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" event={"ID":"011a33f2-468f-4359-9506-c70df101d2a4","Type":"ContainerStarted","Data":"46e1403c8b38f808dddc85013945b34f8e88856dc5d1c6ff61f07060baec753b"} Jul 16 05:05:18.879184 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:18.878788 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:18.896696 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:18.896650 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" podStartSLOduration=2.414287906 podStartE2EDuration="2.896635268s" totalImagesPullingTime="482.347362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:05:18.895389076 +0000 UTC m=+1566.213437646" watchObservedRunningTime="2026-07-16 05:05:18.896635268 +0000 UTC m=+1566.214683855" Jul 16 05:05:19.874966 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:19.874932 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:05:24.889307 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:24.889277 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:27.809555 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:27.809510 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g"] Jul 16 05:05:27.809987 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:27.809725 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" podUID="011a33f2-468f-4359-9506-c70df101d2a4" containerName="maas-api" containerID="cri-o://46e1403c8b38f808dddc85013945b34f8e88856dc5d1c6ff61f07060baec753b" gracePeriod=30 Jul 16 05:05:27.925129 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:27.925101 2585 generic.go:356] "Generic (PLEG): container finished" podID="011a33f2-468f-4359-9506-c70df101d2a4" containerID="46e1403c8b38f808dddc85013945b34f8e88856dc5d1c6ff61f07060baec753b" exitCode=0 Jul 16 05:05:27.925280 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:27.925177 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" event={"ID":"011a33f2-468f-4359-9506-c70df101d2a4","Type":"ContainerDied","Data":"46e1403c8b38f808dddc85013945b34f8e88856dc5d1c6ff61f07060baec753b"} Jul 16 05:05:28.050104 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.050081 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:28.172482 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.172385 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011a33f2-468f-4359-9506-c70df101d2a4-maas-api-tls\") pod \"011a33f2-468f-4359-9506-c70df101d2a4\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " Jul 16 05:05:28.172639 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.172496 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtddf\" (UniqueName: \"kubernetes.io/projected/011a33f2-468f-4359-9506-c70df101d2a4-kube-api-access-mtddf\") pod \"011a33f2-468f-4359-9506-c70df101d2a4\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " Jul 16 05:05:28.172639 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.172532 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011a33f2-468f-4359-9506-c70df101d2a4-openshift-service-ca\") pod \"011a33f2-468f-4359-9506-c70df101d2a4\" (UID: \"011a33f2-468f-4359-9506-c70df101d2a4\") " Jul 16 05:05:28.172941 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.172918 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011a33f2-468f-4359-9506-c70df101d2a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "011a33f2-468f-4359-9506-c70df101d2a4" (UID: "011a33f2-468f-4359-9506-c70df101d2a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 05:05:28.174521 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.174497 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011a33f2-468f-4359-9506-c70df101d2a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "011a33f2-468f-4359-9506-c70df101d2a4" (UID: "011a33f2-468f-4359-9506-c70df101d2a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 05:05:28.174575 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.174547 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011a33f2-468f-4359-9506-c70df101d2a4-kube-api-access-mtddf" (OuterVolumeSpecName: "kube-api-access-mtddf") pod "011a33f2-468f-4359-9506-c70df101d2a4" (UID: "011a33f2-468f-4359-9506-c70df101d2a4"). InnerVolumeSpecName "kube-api-access-mtddf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 05:05:28.273811 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.273759 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtddf\" (UniqueName: \"kubernetes.io/projected/011a33f2-468f-4359-9506-c70df101d2a4-kube-api-access-mtddf\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:05:28.273811 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.273807 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011a33f2-468f-4359-9506-c70df101d2a4-openshift-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:05:28.273982 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.273819 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011a33f2-468f-4359-9506-c70df101d2a4-maas-api-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:05:28.930513 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.930483 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" Jul 16 05:05:28.930934 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.930518 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g" event={"ID":"011a33f2-468f-4359-9506-c70df101d2a4","Type":"ContainerDied","Data":"a0fcd1bb5eeccb3059338aab117314cacd68355bd74581773b65c9a3b020b2eb"} Jul 16 05:05:28.930934 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.930577 2585 scope.go:117] "RemoveContainer" containerID="46e1403c8b38f808dddc85013945b34f8e88856dc5d1c6ff61f07060baec753b" Jul 16 05:05:28.950520 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.950498 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g"] Jul 16 05:05:28.954872 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:28.954851 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9116b405-884b69d6d-nz27g"] Jul 16 05:05:29.208920 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:29.208845 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="011a33f2-468f-4359-9506-c70df101d2a4" path="/var/lib/kubelet/pods/011a33f2-468f-4359-9506-c70df101d2a4/volumes" Jul 16 05:05:29.978699 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:05:29.978661 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:06:55.985858 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:06:55.985821 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:07:03.479572 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:03.479536 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:07:11.882053 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:11.882019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:07:12.564669 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.564627 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl"] Jul 16 05:07:12.565407 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.565386 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011a33f2-468f-4359-9506-c70df101d2a4" containerName="maas-api" Jul 16 05:07:12.565520 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.565409 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="011a33f2-468f-4359-9506-c70df101d2a4" containerName="maas-api" Jul 16 05:07:12.565574 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.565541 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="011a33f2-468f-4359-9506-c70df101d2a4" containerName="maas-api" Jul 16 05:07:12.569551 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.569528 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:12.573029 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.572995 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 05:07:12.573135 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.573019 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ae2b190a-serving-cert\"" Jul 16 05:07:12.573135 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.573095 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47w6t\"" Jul 16 05:07:12.573919 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.573905 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 05:07:12.589627 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.589599 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl"] Jul 16 05:07:12.669715 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.669678 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87ad6bc7-da54-46a2-829c-7d41f8b6741e-maas-api-tls\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:12.669888 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.669734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ad6bc7-da54-46a2-829c-7d41f8b6741e-openshift-service-ca\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:12.669888 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.669810 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7ht\" (UniqueName: \"kubernetes.io/projected/87ad6bc7-da54-46a2-829c-7d41f8b6741e-kube-api-access-hw7ht\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:12.770695 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.770656 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87ad6bc7-da54-46a2-829c-7d41f8b6741e-maas-api-tls\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:12.770869 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.770707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ad6bc7-da54-46a2-829c-7d41f8b6741e-openshift-service-ca\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:12.770869 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.770745 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7ht\" (UniqueName: \"kubernetes.io/projected/87ad6bc7-da54-46a2-829c-7d41f8b6741e-kube-api-access-hw7ht\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:12.770869 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:07:12.770839 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-serving-cert: secret "maas-api-e2e-mt-ae2b190a-serving-cert" not found Jul 16 05:07:12.770982 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:07:12.770918 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ad6bc7-da54-46a2-829c-7d41f8b6741e-maas-api-tls podName:87ad6bc7-da54-46a2-829c-7d41f8b6741e nodeName:}" failed. No retries permitted until 2026-07-16 05:07:13.270900942 +0000 UTC m=+1680.588949478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87ad6bc7-da54-46a2-829c-7d41f8b6741e-maas-api-tls") pod "maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" (UID: "87ad6bc7-da54-46a2-829c-7d41f8b6741e") : secret "maas-api-e2e-mt-ae2b190a-serving-cert" not found Jul 16 05:07:12.771501 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.771483 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ad6bc7-da54-46a2-829c-7d41f8b6741e-openshift-service-ca\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:12.780200 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:12.780181 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7ht\" (UniqueName: \"kubernetes.io/projected/87ad6bc7-da54-46a2-829c-7d41f8b6741e-kube-api-access-hw7ht\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:13.275536 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:13.275508 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87ad6bc7-da54-46a2-829c-7d41f8b6741e-maas-api-tls\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:13.277884 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:13.277863 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87ad6bc7-da54-46a2-829c-7d41f8b6741e-maas-api-tls\") pod \"maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:13.481456 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:13.481422 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47w6t\"" Jul 16 05:07:13.488949 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:13.488924 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:13.628597 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:13.628567 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl"] Jul 16 05:07:13.629920 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:07:13.629896 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ad6bc7_da54_46a2_829c_7d41f8b6741e.slice/crio-71167def0ca43187f3e0de1642f889ff9d5711a3d8e75036a95cc3d2ab86992b WatchSource:0}: Error finding container 71167def0ca43187f3e0de1642f889ff9d5711a3d8e75036a95cc3d2ab86992b: Status 404 returned error can't find the container with id 71167def0ca43187f3e0de1642f889ff9d5711a3d8e75036a95cc3d2ab86992b Jul 16 05:07:14.377136 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:14.377046 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" event={"ID":"87ad6bc7-da54-46a2-829c-7d41f8b6741e","Type":"ContainerStarted","Data":"89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509"} Jul 16 05:07:14.377136 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:14.377096 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" event={"ID":"87ad6bc7-da54-46a2-829c-7d41f8b6741e","Type":"ContainerStarted","Data":"71167def0ca43187f3e0de1642f889ff9d5711a3d8e75036a95cc3d2ab86992b"} Jul 16 05:07:14.377136 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:14.377133 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:14.399493 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:14.399447 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" podStartSLOduration=1.9459451479999998 podStartE2EDuration="2.399429964s" totalImagesPullingTime="453.484816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:07:14.397981566 +0000 UTC m=+1681.716030125" watchObservedRunningTime="2026-07-16 05:07:14.399429964 +0000 UTC m=+1681.717478523" Jul 16 05:07:20.387430 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:20.387400 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:21.895978 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:21.895949 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:07:30.612079 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:30.612043 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl"] Jul 16 05:07:30.612549 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:30.612282 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" podUID="87ad6bc7-da54-46a2-829c-7d41f8b6741e" containerName="maas-api" containerID="cri-o://89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509" gracePeriod=30 Jul 16 05:07:30.861576 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:30.861553 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:30.931130 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:30.931054 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw7ht\" (UniqueName: \"kubernetes.io/projected/87ad6bc7-da54-46a2-829c-7d41f8b6741e-kube-api-access-hw7ht\") pod \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " Jul 16 05:07:30.931130 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:30.931096 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87ad6bc7-da54-46a2-829c-7d41f8b6741e-maas-api-tls\") pod \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " Jul 16 05:07:30.931347 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:30.931182 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ad6bc7-da54-46a2-829c-7d41f8b6741e-openshift-service-ca\") pod \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\" (UID: \"87ad6bc7-da54-46a2-829c-7d41f8b6741e\") " Jul 16 05:07:30.931576 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:30.931544 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ad6bc7-da54-46a2-829c-7d41f8b6741e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87ad6bc7-da54-46a2-829c-7d41f8b6741e" (UID: "87ad6bc7-da54-46a2-829c-7d41f8b6741e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 05:07:30.933242 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:30.933216 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ad6bc7-da54-46a2-829c-7d41f8b6741e-kube-api-access-hw7ht" (OuterVolumeSpecName: "kube-api-access-hw7ht") pod "87ad6bc7-da54-46a2-829c-7d41f8b6741e" (UID: "87ad6bc7-da54-46a2-829c-7d41f8b6741e"). InnerVolumeSpecName "kube-api-access-hw7ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 05:07:30.933316 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:30.933252 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ad6bc7-da54-46a2-829c-7d41f8b6741e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87ad6bc7-da54-46a2-829c-7d41f8b6741e" (UID: "87ad6bc7-da54-46a2-829c-7d41f8b6741e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 05:07:31.032486 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.032452 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ad6bc7-da54-46a2-829c-7d41f8b6741e-openshift-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:07:31.032486 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.032480 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw7ht\" (UniqueName: \"kubernetes.io/projected/87ad6bc7-da54-46a2-829c-7d41f8b6741e-kube-api-access-hw7ht\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:07:31.032678 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.032495 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87ad6bc7-da54-46a2-829c-7d41f8b6741e-maas-api-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:07:31.452656 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.452620 2585 generic.go:356] "Generic (PLEG): container finished" podID="87ad6bc7-da54-46a2-829c-7d41f8b6741e" containerID="89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509" exitCode=0 Jul 16 05:07:31.452878 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.452684 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" Jul 16 05:07:31.452878 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.452701 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" event={"ID":"87ad6bc7-da54-46a2-829c-7d41f8b6741e","Type":"ContainerDied","Data":"89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509"} Jul 16 05:07:31.452878 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.452755 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl" event={"ID":"87ad6bc7-da54-46a2-829c-7d41f8b6741e","Type":"ContainerDied","Data":"71167def0ca43187f3e0de1642f889ff9d5711a3d8e75036a95cc3d2ab86992b"} Jul 16 05:07:31.452878 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.452799 2585 scope.go:117] "RemoveContainer" containerID="89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509" Jul 16 05:07:31.460963 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.460945 2585 scope.go:117] "RemoveContainer" containerID="89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509" Jul 16 05:07:31.461210 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:07:31.461193 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509\": container with ID starting with 89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509 not found: ID does not exist" containerID="89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509" Jul 16 05:07:31.461273 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.461217 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509"} err="failed to get container status \"89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509\": rpc error: code = NotFound desc = could not find container \"89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509\": container with ID starting with 89845b0afa6c26394160a615d0417f30fa18d8bd92b53ff7a7319d2010f1d509 not found: ID does not exist" Jul 16 05:07:31.469135 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.469088 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl"] Jul 16 05:07:31.474237 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:31.474217 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae2b190a-646fffdb5-fnzbl"] Jul 16 05:07:33.209136 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:33.209105 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ad6bc7-da54-46a2-829c-7d41f8b6741e" path="/var/lib/kubelet/pods/87ad6bc7-da54-46a2-829c-7d41f8b6741e/volumes" Jul 16 05:07:33.674759 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:07:33.674725 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:08:03.385351 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:03.385320 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:08:15.978137 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:15.978099 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:08:54.917554 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.917516 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk"] Jul 16 05:08:54.918068 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.917852 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ad6bc7-da54-46a2-829c-7d41f8b6741e" containerName="maas-api" Jul 16 05:08:54.918068 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.917863 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ad6bc7-da54-46a2-829c-7d41f8b6741e" containerName="maas-api" Jul 16 05:08:54.918068 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.917924 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ad6bc7-da54-46a2-829c-7d41f8b6741e" containerName="maas-api" Jul 16 05:08:54.920829 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.920811 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:54.923257 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.923234 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 05:08:54.923380 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.923318 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-82a682-serving-cert\"" Jul 16 05:08:54.923475 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.923457 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 05:08:54.923606 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.923592 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47w6t\"" Jul 16 05:08:54.930321 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.930299 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk"] Jul 16 05:08:54.940336 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.940309 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07151b10-d023-4562-a083-13bc4e9bfa6f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:54.940459 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.940443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07151b10-d023-4562-a083-13bc4e9bfa6f-maas-api-tls\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:54.940517 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:54.940483 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krw5j\" (UniqueName: \"kubernetes.io/projected/07151b10-d023-4562-a083-13bc4e9bfa6f-kube-api-access-krw5j\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:55.041613 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:55.041578 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07151b10-d023-4562-a083-13bc4e9bfa6f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:55.041834 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:55.041629 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07151b10-d023-4562-a083-13bc4e9bfa6f-maas-api-tls\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:55.041834 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:55.041667 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krw5j\" (UniqueName: \"kubernetes.io/projected/07151b10-d023-4562-a083-13bc4e9bfa6f-kube-api-access-krw5j\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:55.041978 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:08:55.041834 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-serving-cert: secret "maas-api-e2e-inf-a-82a682-serving-cert" not found Jul 16 05:08:55.041978 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:08:55.041928 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07151b10-d023-4562-a083-13bc4e9bfa6f-maas-api-tls podName:07151b10-d023-4562-a083-13bc4e9bfa6f nodeName:}" failed. No retries permitted until 2026-07-16 05:08:55.541904869 +0000 UTC m=+1782.859953419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07151b10-d023-4562-a083-13bc4e9bfa6f-maas-api-tls") pod "maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" (UID: "07151b10-d023-4562-a083-13bc4e9bfa6f") : secret "maas-api-e2e-inf-a-82a682-serving-cert" not found Jul 16 05:08:55.042326 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:55.042305 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07151b10-d023-4562-a083-13bc4e9bfa6f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:55.050814 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:55.050788 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krw5j\" (UniqueName: \"kubernetes.io/projected/07151b10-d023-4562-a083-13bc4e9bfa6f-kube-api-access-krw5j\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:55.546360 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:55.546324 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07151b10-d023-4562-a083-13bc4e9bfa6f-maas-api-tls\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:55.548681 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:55.548652 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07151b10-d023-4562-a083-13bc4e9bfa6f-maas-api-tls\") pod \"maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:55.832179 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:55.832098 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:55.965097 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:55.965017 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk"] Jul 16 05:08:55.967862 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:08:55.967834 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07151b10_d023_4562_a083_13bc4e9bfa6f.slice/crio-a92f1372917a669b7f625151e1996e9bc962b0537cad772ec63453f292eaf0dd WatchSource:0}: Error finding container a92f1372917a669b7f625151e1996e9bc962b0537cad772ec63453f292eaf0dd: Status 404 returned error can't find the container with id a92f1372917a669b7f625151e1996e9bc962b0537cad772ec63453f292eaf0dd Jul 16 05:08:56.679231 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:56.679193 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:08:56.818992 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:56.818890 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" event={"ID":"07151b10-d023-4562-a083-13bc4e9bfa6f","Type":"ContainerStarted","Data":"c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1"} Jul 16 05:08:56.818992 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:56.818943 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" event={"ID":"07151b10-d023-4562-a083-13bc4e9bfa6f","Type":"ContainerStarted","Data":"a92f1372917a669b7f625151e1996e9bc962b0537cad772ec63453f292eaf0dd"} Jul 16 05:08:56.819172 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:56.818991 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:08:56.837401 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:08:56.837355 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" podStartSLOduration=2.260541581 podStartE2EDuration="2.837338216s" totalImagesPullingTime="576.796635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:08:56.835548983 +0000 UTC m=+1784.153597600" watchObservedRunningTime="2026-07-16 05:08:56.837338216 +0000 UTC m=+1784.155386767" Jul 16 05:09:02.830615 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:02.830588 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:09:02.917115 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:02.917084 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26"] Jul 16 05:09:02.920667 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:02.920649 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:02.923275 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:02.923255 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-05c504-serving-cert\"" Jul 16 05:09:02.933884 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:02.933856 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26"] Jul 16 05:09:03.017199 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.017164 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.017367 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.017260 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4hd\" (UniqueName: \"kubernetes.io/projected/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-kube-api-access-2g4hd\") pod \"maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.017367 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.017294 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-maas-api-tls\") pod \"maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.118238 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.118162 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4hd\" (UniqueName: \"kubernetes.io/projected/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-kube-api-access-2g4hd\") pod \"maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.118238 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.118216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-maas-api-tls\") pod \"maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.118452 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.118249 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.118933 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.118910 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.120782 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.120746 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-maas-api-tls\") pod \"maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.137067 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.137039 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4hd\" (UniqueName: \"kubernetes.io/projected/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-kube-api-access-2g4hd\") pod \"maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.230845 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.230810 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:03.361869 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.361836 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26"] Jul 16 05:09:03.365063 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:09:03.365035 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2921ae_1d72_48b3_9a54_18fe12c4bbf1.slice/crio-33a7d1b685f5ca69d544a97e2ff8fe4a435b479a8cba2fccbb5ec4c3c53a76d9 WatchSource:0}: Error finding container 33a7d1b685f5ca69d544a97e2ff8fe4a435b479a8cba2fccbb5ec4c3c53a76d9: Status 404 returned error can't find the container with id 33a7d1b685f5ca69d544a97e2ff8fe4a435b479a8cba2fccbb5ec4c3c53a76d9 Jul 16 05:09:03.850327 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:03.850287 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" event={"ID":"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1","Type":"ContainerStarted","Data":"33a7d1b685f5ca69d544a97e2ff8fe4a435b479a8cba2fccbb5ec4c3c53a76d9"} Jul 16 05:09:04.856322 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:04.856285 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" event={"ID":"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1","Type":"ContainerStarted","Data":"4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c"} Jul 16 05:09:04.856745 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:04.856351 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:04.875018 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:04.874972 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" podStartSLOduration=2.179934179 podStartE2EDuration="2.874954945s" totalImagesPullingTime="695.020766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:09:04.873214222 +0000 UTC m=+1792.191262800" watchObservedRunningTime="2026-07-16 05:09:04.874954945 +0000 UTC m=+1792.193003503" Jul 16 05:09:10.867393 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:10.867305 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:09:10.974823 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:10.974785 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:09:13.238372 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:13.238343 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 05:09:13.240570 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:13.240550 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 05:09:13.245471 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:13.245454 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 05:09:13.247397 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:13.247381 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 05:09:20.290059 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:20.290020 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:09:31.381521 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:31.381485 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:09:45.686094 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:45.686059 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:09:59.972882 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:09:59.972845 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:10:12.383245 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:12.383208 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:10:44.639833 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:44.639759 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk"] Jul 16 05:10:44.640389 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:44.639988 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" podUID="07151b10-d023-4562-a083-13bc4e9bfa6f" containerName="maas-api" containerID="cri-o://c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1" gracePeriod=30 Jul 16 05:10:44.890614 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:44.890560 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:10:44.979020 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:44.978988 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07151b10-d023-4562-a083-13bc4e9bfa6f-openshift-service-ca\") pod \"07151b10-d023-4562-a083-13bc4e9bfa6f\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " Jul 16 05:10:44.979193 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:44.979052 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07151b10-d023-4562-a083-13bc4e9bfa6f-maas-api-tls\") pod \"07151b10-d023-4562-a083-13bc4e9bfa6f\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " Jul 16 05:10:44.979193 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:44.979069 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krw5j\" (UniqueName: \"kubernetes.io/projected/07151b10-d023-4562-a083-13bc4e9bfa6f-kube-api-access-krw5j\") pod \"07151b10-d023-4562-a083-13bc4e9bfa6f\" (UID: \"07151b10-d023-4562-a083-13bc4e9bfa6f\") " Jul 16 05:10:44.979339 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:44.979313 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07151b10-d023-4562-a083-13bc4e9bfa6f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07151b10-d023-4562-a083-13bc4e9bfa6f" (UID: "07151b10-d023-4562-a083-13bc4e9bfa6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 05:10:44.981202 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:44.981178 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07151b10-d023-4562-a083-13bc4e9bfa6f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07151b10-d023-4562-a083-13bc4e9bfa6f" (UID: "07151b10-d023-4562-a083-13bc4e9bfa6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 05:10:44.981290 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:44.981177 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07151b10-d023-4562-a083-13bc4e9bfa6f-kube-api-access-krw5j" (OuterVolumeSpecName: "kube-api-access-krw5j") pod "07151b10-d023-4562-a083-13bc4e9bfa6f" (UID: "07151b10-d023-4562-a083-13bc4e9bfa6f"). InnerVolumeSpecName "kube-api-access-krw5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 05:10:45.080212 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.080181 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07151b10-d023-4562-a083-13bc4e9bfa6f-maas-api-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:10:45.080212 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.080208 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krw5j\" (UniqueName: \"kubernetes.io/projected/07151b10-d023-4562-a083-13bc4e9bfa6f-kube-api-access-krw5j\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:10:45.080212 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.080217 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07151b10-d023-4562-a083-13bc4e9bfa6f-openshift-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:10:45.295043 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.295005 2585 generic.go:356] "Generic (PLEG): container finished" podID="07151b10-d023-4562-a083-13bc4e9bfa6f" containerID="c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1" exitCode=0 Jul 16 05:10:45.295043 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.295043 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" Jul 16 05:10:45.295286 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.295091 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" event={"ID":"07151b10-d023-4562-a083-13bc4e9bfa6f","Type":"ContainerDied","Data":"c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1"} Jul 16 05:10:45.295286 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.295152 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk" event={"ID":"07151b10-d023-4562-a083-13bc4e9bfa6f","Type":"ContainerDied","Data":"a92f1372917a669b7f625151e1996e9bc962b0537cad772ec63453f292eaf0dd"} Jul 16 05:10:45.295286 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.295181 2585 scope.go:117] "RemoveContainer" containerID="c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1" Jul 16 05:10:45.303113 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.303096 2585 scope.go:117] "RemoveContainer" containerID="c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1" Jul 16 05:10:45.303383 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:10:45.303368 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1\": container with ID starting with c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1 not found: ID does not exist" containerID="c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1" Jul 16 05:10:45.303425 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.303394 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1"} err="failed to get container status \"c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1\": rpc error: code = NotFound desc = could not find container \"c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1\": container with ID starting with c9ef9670add3f7f9821a29a767b384bb9df0a1dd62ee82d1f9eed4087dcc74f1 not found: ID does not exist" Jul 16 05:10:45.326636 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.326613 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk"] Jul 16 05:10:45.331892 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:45.331869 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82a682-64bb4d67f7-mp5jk"] Jul 16 05:10:47.208973 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:47.208943 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07151b10-d023-4562-a083-13bc4e9bfa6f" path="/var/lib/kubelet/pods/07151b10-d023-4562-a083-13bc4e9bfa6f/volumes" Jul 16 05:10:48.081725 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:48.081690 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:10:58.697044 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:58.696998 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26"] Jul 16 05:10:58.697636 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:58.697257 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" podUID="ed2921ae-1d72-48b3-9a54-18fe12c4bbf1" containerName="maas-api" containerID="cri-o://4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c" gracePeriod=30 Jul 16 05:10:58.950341 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:58.950289 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:10:59.000736 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.000708 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g4hd\" (UniqueName: \"kubernetes.io/projected/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-kube-api-access-2g4hd\") pod \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " Jul 16 05:10:59.000894 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.000752 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-openshift-service-ca\") pod \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " Jul 16 05:10:59.000894 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.000788 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-maas-api-tls\") pod \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\" (UID: \"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1\") " Jul 16 05:10:59.001091 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.001067 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed2921ae-1d72-48b3-9a54-18fe12c4bbf1" (UID: "ed2921ae-1d72-48b3-9a54-18fe12c4bbf1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 05:10:59.002838 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.002812 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed2921ae-1d72-48b3-9a54-18fe12c4bbf1" (UID: "ed2921ae-1d72-48b3-9a54-18fe12c4bbf1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 05:10:59.003019 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.002878 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-kube-api-access-2g4hd" (OuterVolumeSpecName: "kube-api-access-2g4hd") pod "ed2921ae-1d72-48b3-9a54-18fe12c4bbf1" (UID: "ed2921ae-1d72-48b3-9a54-18fe12c4bbf1"). InnerVolumeSpecName "kube-api-access-2g4hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 05:10:59.102082 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.102041 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g4hd\" (UniqueName: \"kubernetes.io/projected/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-kube-api-access-2g4hd\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:10:59.102082 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.102075 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-openshift-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:10:59.102082 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.102087 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1-maas-api-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:10:59.356685 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.356645 2585 generic.go:356] "Generic (PLEG): container finished" podID="ed2921ae-1d72-48b3-9a54-18fe12c4bbf1" containerID="4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c" exitCode=0 Jul 16 05:10:59.356685 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.356683 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" Jul 16 05:10:59.356908 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.356722 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" event={"ID":"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1","Type":"ContainerDied","Data":"4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c"} Jul 16 05:10:59.356908 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.356800 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26" event={"ID":"ed2921ae-1d72-48b3-9a54-18fe12c4bbf1","Type":"ContainerDied","Data":"33a7d1b685f5ca69d544a97e2ff8fe4a435b479a8cba2fccbb5ec4c3c53a76d9"} Jul 16 05:10:59.356908 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.356819 2585 scope.go:117] "RemoveContainer" containerID="4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c" Jul 16 05:10:59.365084 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.365067 2585 scope.go:117] "RemoveContainer" containerID="4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c" Jul 16 05:10:59.365342 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:10:59.365324 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c\": container with ID starting with 4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c not found: ID does not exist" containerID="4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c" Jul 16 05:10:59.365439 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.365350 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c"} err="failed to get container status \"4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c\": rpc error: code = NotFound desc = could not find container \"4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c\": container with ID starting with 4f60b2480eb52517c3f545ba9e45ce552524169df0c5d9866c0432c0fd09fb2c not found: ID does not exist" Jul 16 05:10:59.374702 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.374681 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26"] Jul 16 05:10:59.377640 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:10:59.377621 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-05c504-6c86c56fbb-b2p26"] Jul 16 05:11:01.209583 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:01.209548 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2921ae-1d72-48b3-9a54-18fe12c4bbf1" path="/var/lib/kubelet/pods/ed2921ae-1d72-48b3-9a54-18fe12c4bbf1/volumes" Jul 16 05:11:02.904315 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:02.904283 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:11:07.623978 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.623939 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k"] Jul 16 05:11:07.624361 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.624262 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2921ae-1d72-48b3-9a54-18fe12c4bbf1" containerName="maas-api" Jul 16 05:11:07.624361 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.624274 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2921ae-1d72-48b3-9a54-18fe12c4bbf1" containerName="maas-api" Jul 16 05:11:07.624361 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.624298 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07151b10-d023-4562-a083-13bc4e9bfa6f" containerName="maas-api" Jul 16 05:11:07.624361 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.624304 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="07151b10-d023-4562-a083-13bc4e9bfa6f" containerName="maas-api" Jul 16 05:11:07.624361 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.624353 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2921ae-1d72-48b3-9a54-18fe12c4bbf1" containerName="maas-api" Jul 16 05:11:07.624361 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.624362 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="07151b10-d023-4562-a083-13bc4e9bfa6f" containerName="maas-api" Jul 16 05:11:07.628690 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.628665 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.631171 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.631145 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47w6t\"" Jul 16 05:11:07.631309 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.631173 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e9b55e-serving-cert\"" Jul 16 05:11:07.631920 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.631904 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 05:11:07.631970 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.631952 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 05:11:07.640624 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.640598 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k"] Jul 16 05:11:07.780040 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.780006 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c9d7e6-8817-455a-8070-62cd12845356-openshift-service-ca\") pod \"maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.780211 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.780079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c9d7e6-8817-455a-8070-62cd12845356-maas-api-tls\") pod \"maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.780211 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.780114 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n857\" (UniqueName: \"kubernetes.io/projected/e0c9d7e6-8817-455a-8070-62cd12845356-kube-api-access-4n857\") pod \"maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.880672 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.880587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c9d7e6-8817-455a-8070-62cd12845356-openshift-service-ca\") pod \"maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.880844 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.880671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c9d7e6-8817-455a-8070-62cd12845356-maas-api-tls\") pod \"maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.880844 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.880703 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n857\" (UniqueName: \"kubernetes.io/projected/e0c9d7e6-8817-455a-8070-62cd12845356-kube-api-access-4n857\") pod \"maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.881489 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.881459 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c9d7e6-8817-455a-8070-62cd12845356-openshift-service-ca\") pod \"maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.883251 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.883222 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c9d7e6-8817-455a-8070-62cd12845356-maas-api-tls\") pod \"maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.890115 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.890091 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n857\" (UniqueName: \"kubernetes.io/projected/e0c9d7e6-8817-455a-8070-62cd12845356-kube-api-access-4n857\") pod \"maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:07.939407 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:07.939371 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:08.072152 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:08.072124 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k"] Jul 16 05:11:08.074570 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:11:08.074539 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c9d7e6_8817_455a_8070_62cd12845356.slice/crio-fe62cf327f6922a29da7bf0dfa99066cbba5604105b2d0ef80ec253426e15f6d WatchSource:0}: Error finding container fe62cf327f6922a29da7bf0dfa99066cbba5604105b2d0ef80ec253426e15f6d: Status 404 returned error can't find the container with id fe62cf327f6922a29da7bf0dfa99066cbba5604105b2d0ef80ec253426e15f6d Jul 16 05:11:08.075934 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:08.075916 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 05:11:08.396561 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:08.396529 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" event={"ID":"e0c9d7e6-8817-455a-8070-62cd12845356","Type":"ContainerStarted","Data":"fe62cf327f6922a29da7bf0dfa99066cbba5604105b2d0ef80ec253426e15f6d"} Jul 16 05:11:09.402469 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:09.402438 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" event={"ID":"e0c9d7e6-8817-455a-8070-62cd12845356","Type":"ContainerStarted","Data":"c7a0b080c250783449caa659c8c0489d8ce9006eb19874eae3e5d82cd6aa9124"} Jul 16 05:11:09.402819 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:09.402504 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:09.421836 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:09.421736 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" podStartSLOduration=2.012750839 podStartE2EDuration="2.421716815s" totalImagesPullingTime="408.965976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:11:09.420572815 +0000 UTC m=+1916.738621599" watchObservedRunningTime="2026-07-16 05:11:09.421716815 +0000 UTC m=+1916.739765375" Jul 16 05:11:13.583119 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:13.583044 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:11:15.412029 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.411998 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:15.545400 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.545362 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr"] Jul 16 05:11:15.550025 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.550004 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.552279 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.552252 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bd340d-serving-cert\"" Jul 16 05:11:15.559783 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.559742 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr"] Jul 16 05:11:15.646759 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.646726 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e4662e-e70b-40e9-bf18-c1c2293ef67f-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.647030 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.646840 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e4662e-e70b-40e9-bf18-c1c2293ef67f-maas-api-tls\") pod \"maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.647030 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.646965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54sv6\" (UniqueName: \"kubernetes.io/projected/05e4662e-e70b-40e9-bf18-c1c2293ef67f-kube-api-access-54sv6\") pod \"maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.748206 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.748114 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e4662e-e70b-40e9-bf18-c1c2293ef67f-maas-api-tls\") pod \"maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.748206 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.748179 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54sv6\" (UniqueName: \"kubernetes.io/projected/05e4662e-e70b-40e9-bf18-c1c2293ef67f-kube-api-access-54sv6\") pod \"maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.748448 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.748211 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e4662e-e70b-40e9-bf18-c1c2293ef67f-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.748936 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.748908 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e4662e-e70b-40e9-bf18-c1c2293ef67f-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.751016 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.750997 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e4662e-e70b-40e9-bf18-c1c2293ef67f-maas-api-tls\") pod \"maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.756242 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.756226 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54sv6\" (UniqueName: \"kubernetes.io/projected/05e4662e-e70b-40e9-bf18-c1c2293ef67f-kube-api-access-54sv6\") pod \"maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.861179 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.861141 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:15.993443 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:15.993419 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr"] Jul 16 05:11:15.995633 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:11:15.995601 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e4662e_e70b_40e9_bf18_c1c2293ef67f.slice/crio-e42d8298690a7b42545685fc72812c326f891356d8cf9be1d9f721b5606b3b13 WatchSource:0}: Error finding container e42d8298690a7b42545685fc72812c326f891356d8cf9be1d9f721b5606b3b13: Status 404 returned error can't find the container with id e42d8298690a7b42545685fc72812c326f891356d8cf9be1d9f721b5606b3b13 Jul 16 05:11:16.434614 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:16.434580 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" event={"ID":"05e4662e-e70b-40e9-bf18-c1c2293ef67f","Type":"ContainerStarted","Data":"cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c"} Jul 16 05:11:16.435048 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:16.434640 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" event={"ID":"05e4662e-e70b-40e9-bf18-c1c2293ef67f","Type":"ContainerStarted","Data":"e42d8298690a7b42545685fc72812c326f891356d8cf9be1d9f721b5606b3b13"} Jul 16 05:11:16.435048 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:16.434670 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:16.452255 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:16.452209 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" podStartSLOduration=1.101094041 podStartE2EDuration="1.452193499s" totalImagesPullingTime="351.099458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:11:16.450913025 +0000 UTC m=+1923.768961585" watchObservedRunningTime="2026-07-16 05:11:16.452193499 +0000 UTC m=+1923.770242101" Jul 16 05:11:22.446589 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:22.446559 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:26.495025 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:26.494994 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:11:36.345314 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.345274 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k"] Jul 16 05:11:36.345821 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.345519 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" podUID="e0c9d7e6-8817-455a-8070-62cd12845356" containerName="maas-api" containerID="cri-o://c7a0b080c250783449caa659c8c0489d8ce9006eb19874eae3e5d82cd6aa9124" gracePeriod=30 Jul 16 05:11:36.528239 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.528202 2585 generic.go:356] "Generic (PLEG): container finished" podID="e0c9d7e6-8817-455a-8070-62cd12845356" containerID="c7a0b080c250783449caa659c8c0489d8ce9006eb19874eae3e5d82cd6aa9124" exitCode=0 Jul 16 05:11:36.528398 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.528260 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" event={"ID":"e0c9d7e6-8817-455a-8070-62cd12845356","Type":"ContainerDied","Data":"c7a0b080c250783449caa659c8c0489d8ce9006eb19874eae3e5d82cd6aa9124"} Jul 16 05:11:36.597248 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.597186 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:36.623260 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.623232 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c9d7e6-8817-455a-8070-62cd12845356-openshift-service-ca\") pod \"e0c9d7e6-8817-455a-8070-62cd12845356\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " Jul 16 05:11:36.623441 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.623283 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n857\" (UniqueName: \"kubernetes.io/projected/e0c9d7e6-8817-455a-8070-62cd12845356-kube-api-access-4n857\") pod \"e0c9d7e6-8817-455a-8070-62cd12845356\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " Jul 16 05:11:36.623441 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.623330 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c9d7e6-8817-455a-8070-62cd12845356-maas-api-tls\") pod \"e0c9d7e6-8817-455a-8070-62cd12845356\" (UID: \"e0c9d7e6-8817-455a-8070-62cd12845356\") " Jul 16 05:11:36.623590 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.623567 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c9d7e6-8817-455a-8070-62cd12845356-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0c9d7e6-8817-455a-8070-62cd12845356" (UID: "e0c9d7e6-8817-455a-8070-62cd12845356"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 05:11:36.625409 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.625377 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c9d7e6-8817-455a-8070-62cd12845356-kube-api-access-4n857" (OuterVolumeSpecName: "kube-api-access-4n857") pod "e0c9d7e6-8817-455a-8070-62cd12845356" (UID: "e0c9d7e6-8817-455a-8070-62cd12845356"). InnerVolumeSpecName "kube-api-access-4n857". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 05:11:36.625512 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.625406 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c9d7e6-8817-455a-8070-62cd12845356-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0c9d7e6-8817-455a-8070-62cd12845356" (UID: "e0c9d7e6-8817-455a-8070-62cd12845356"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 05:11:36.724200 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.724164 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c9d7e6-8817-455a-8070-62cd12845356-maas-api-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:11:36.724200 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.724194 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c9d7e6-8817-455a-8070-62cd12845356-openshift-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:11:36.724200 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:36.724205 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n857\" (UniqueName: \"kubernetes.io/projected/e0c9d7e6-8817-455a-8070-62cd12845356-kube-api-access-4n857\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:11:37.534211 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:37.534126 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" Jul 16 05:11:37.534211 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:37.534152 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k" event={"ID":"e0c9d7e6-8817-455a-8070-62cd12845356","Type":"ContainerDied","Data":"fe62cf327f6922a29da7bf0dfa99066cbba5604105b2d0ef80ec253426e15f6d"} Jul 16 05:11:37.534644 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:37.534216 2585 scope.go:117] "RemoveContainer" containerID="c7a0b080c250783449caa659c8c0489d8ce9006eb19874eae3e5d82cd6aa9124" Jul 16 05:11:37.550170 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:37.550134 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k"] Jul 16 05:11:37.551966 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:37.551946 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e9b55e-56d465c47d-x7v7k"] Jul 16 05:11:39.209182 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:39.209147 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c9d7e6-8817-455a-8070-62cd12845356" path="/var/lib/kubelet/pods/e0c9d7e6-8817-455a-8070-62cd12845356/volumes" Jul 16 05:11:41.181734 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:41.181689 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:11:49.535127 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.535087 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr"] Jul 16 05:11:49.535736 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.535361 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" podUID="05e4662e-e70b-40e9-bf18-c1c2293ef67f" containerName="maas-api" containerID="cri-o://cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c" gracePeriod=30 Jul 16 05:11:49.778963 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.778937 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:49.820046 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.820016 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54sv6\" (UniqueName: \"kubernetes.io/projected/05e4662e-e70b-40e9-bf18-c1c2293ef67f-kube-api-access-54sv6\") pod \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " Jul 16 05:11:49.820222 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.820062 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e4662e-e70b-40e9-bf18-c1c2293ef67f-maas-api-tls\") pod \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " Jul 16 05:11:49.820222 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.820121 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e4662e-e70b-40e9-bf18-c1c2293ef67f-openshift-service-ca\") pod \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\" (UID: \"05e4662e-e70b-40e9-bf18-c1c2293ef67f\") " Jul 16 05:11:49.820536 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.820512 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e4662e-e70b-40e9-bf18-c1c2293ef67f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05e4662e-e70b-40e9-bf18-c1c2293ef67f" (UID: "05e4662e-e70b-40e9-bf18-c1c2293ef67f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 05:11:49.822254 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.822229 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05e4662e-e70b-40e9-bf18-c1c2293ef67f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05e4662e-e70b-40e9-bf18-c1c2293ef67f" (UID: "05e4662e-e70b-40e9-bf18-c1c2293ef67f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 05:11:49.822372 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.822233 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e4662e-e70b-40e9-bf18-c1c2293ef67f-kube-api-access-54sv6" (OuterVolumeSpecName: "kube-api-access-54sv6") pod "05e4662e-e70b-40e9-bf18-c1c2293ef67f" (UID: "05e4662e-e70b-40e9-bf18-c1c2293ef67f"). InnerVolumeSpecName "kube-api-access-54sv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 05:11:49.921397 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.921354 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e4662e-e70b-40e9-bf18-c1c2293ef67f-maas-api-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:11:49.921397 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.921389 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e4662e-e70b-40e9-bf18-c1c2293ef67f-openshift-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:11:49.921397 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:49.921402 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54sv6\" (UniqueName: \"kubernetes.io/projected/05e4662e-e70b-40e9-bf18-c1c2293ef67f-kube-api-access-54sv6\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:11:50.593731 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:50.593691 2585 generic.go:356] "Generic (PLEG): container finished" podID="05e4662e-e70b-40e9-bf18-c1c2293ef67f" containerID="cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c" exitCode=0 Jul 16 05:11:50.593731 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:50.593732 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" Jul 16 05:11:50.594215 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:50.593791 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" event={"ID":"05e4662e-e70b-40e9-bf18-c1c2293ef67f","Type":"ContainerDied","Data":"cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c"} Jul 16 05:11:50.594215 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:50.593846 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr" event={"ID":"05e4662e-e70b-40e9-bf18-c1c2293ef67f","Type":"ContainerDied","Data":"e42d8298690a7b42545685fc72812c326f891356d8cf9be1d9f721b5606b3b13"} Jul 16 05:11:50.594215 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:50.593869 2585 scope.go:117] "RemoveContainer" containerID="cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c" Jul 16 05:11:50.602618 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:50.602602 2585 scope.go:117] "RemoveContainer" containerID="cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c" Jul 16 05:11:50.602902 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:11:50.602880 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c\": container with ID starting with cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c not found: ID does not exist" containerID="cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c" Jul 16 05:11:50.602964 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:50.602910 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c"} err="failed to get container status \"cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c\": rpc error: code = NotFound desc = could not find container \"cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c\": container with ID starting with cbdfd803366636ff55acbcb8284516afe5c1a5f6ef7864de036e7046f953839c not found: ID does not exist" Jul 16 05:11:50.614230 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:50.614205 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr"] Jul 16 05:11:50.616531 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:50.616511 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd340d-5bcf4d5d5-7dsgr"] Jul 16 05:11:51.209366 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:51.209334 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05e4662e-e70b-40e9-bf18-c1c2293ef67f" path="/var/lib/kubelet/pods/05e4662e-e70b-40e9-bf18-c1c2293ef67f/volumes" Jul 16 05:11:53.676238 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:53.676200 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:11:59.899920 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.899889 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8"] Jul 16 05:11:59.900392 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.900373 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c9d7e6-8817-455a-8070-62cd12845356" containerName="maas-api" Jul 16 05:11:59.900489 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.900395 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c9d7e6-8817-455a-8070-62cd12845356" containerName="maas-api" Jul 16 05:11:59.900489 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.900413 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e4662e-e70b-40e9-bf18-c1c2293ef67f" containerName="maas-api" Jul 16 05:11:59.900489 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.900423 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e4662e-e70b-40e9-bf18-c1c2293ef67f" containerName="maas-api" Jul 16 05:11:59.900631 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.900530 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c9d7e6-8817-455a-8070-62cd12845356" containerName="maas-api" Jul 16 05:11:59.900631 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.900549 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e4662e-e70b-40e9-bf18-c1c2293ef67f" containerName="maas-api" Jul 16 05:11:59.904651 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.904630 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:11:59.907389 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.907363 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 05:11:59.907489 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.907398 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 05:11:59.907489 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.907400 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a01a82-serving-cert\"" Jul 16 05:11:59.907938 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.907924 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47w6t\"" Jul 16 05:11:59.914111 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:11:59.914088 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8"] Jul 16 05:12:00.006890 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.006850 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba53e3ad-216b-4347-ac5c-5816f378116f-maas-api-tls\") pod \"maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.007071 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.006987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53e3ad-216b-4347-ac5c-5816f378116f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.007071 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.007031 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvkwv\" (UniqueName: \"kubernetes.io/projected/ba53e3ad-216b-4347-ac5c-5816f378116f-kube-api-access-nvkwv\") pod \"maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.107434 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.107401 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53e3ad-216b-4347-ac5c-5816f378116f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.107434 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.107437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvkwv\" (UniqueName: \"kubernetes.io/projected/ba53e3ad-216b-4347-ac5c-5816f378116f-kube-api-access-nvkwv\") pod \"maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.107661 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.107460 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba53e3ad-216b-4347-ac5c-5816f378116f-maas-api-tls\") pod \"maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.108099 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.108063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53e3ad-216b-4347-ac5c-5816f378116f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.109744 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.109724 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba53e3ad-216b-4347-ac5c-5816f378116f-maas-api-tls\") pod \"maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.116807 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.116755 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvkwv\" (UniqueName: \"kubernetes.io/projected/ba53e3ad-216b-4347-ac5c-5816f378116f-kube-api-access-nvkwv\") pod \"maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.215715 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.215610 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:00.367935 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.367901 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8"] Jul 16 05:12:00.370232 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:12:00.370202 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba53e3ad_216b_4347_ac5c_5816f378116f.slice/crio-3ff87c7234c6368ce144ae3b8166f42ad527ce58223c0859a1c0f1d983400582 WatchSource:0}: Error finding container 3ff87c7234c6368ce144ae3b8166f42ad527ce58223c0859a1c0f1d983400582: Status 404 returned error can't find the container with id 3ff87c7234c6368ce144ae3b8166f42ad527ce58223c0859a1c0f1d983400582 Jul 16 05:12:00.639087 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:00.639050 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" event={"ID":"ba53e3ad-216b-4347-ac5c-5816f378116f","Type":"ContainerStarted","Data":"3ff87c7234c6368ce144ae3b8166f42ad527ce58223c0859a1c0f1d983400582"} Jul 16 05:12:01.645380 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:01.645341 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" event={"ID":"ba53e3ad-216b-4347-ac5c-5816f378116f","Type":"ContainerStarted","Data":"77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011"} Jul 16 05:12:01.645807 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:01.645407 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:01.662745 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:01.662698 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" podStartSLOduration=2.32860571 podStartE2EDuration="2.662681804s" totalImagesPullingTime="334.076094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:12:01.661526426 +0000 UTC m=+1968.979574977" watchObservedRunningTime="2026-07-16 05:12:01.662681804 +0000 UTC m=+1968.980730341" Jul 16 05:12:03.699380 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:03.699350 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:12:07.654302 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:07.654225 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:12:08.062751 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.062713 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd"] Jul 16 05:12:08.066790 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.066756 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.070264 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.070242 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-12efc5-serving-cert\"" Jul 16 05:12:08.086316 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.086292 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd"] Jul 16 05:12:08.180779 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.180744 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77287066-a06b-4f29-941a-fe165f9ee72b-maas-api-tls\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.180945 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.180846 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48n9n\" (UniqueName: \"kubernetes.io/projected/77287066-a06b-4f29-941a-fe165f9ee72b-kube-api-access-48n9n\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.180945 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.180911 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77287066-a06b-4f29-941a-fe165f9ee72b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.282169 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.282131 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77287066-a06b-4f29-941a-fe165f9ee72b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.282347 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.282192 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77287066-a06b-4f29-941a-fe165f9ee72b-maas-api-tls\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.282347 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:12:08.282273 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-serving-cert: secret "maas-api-e2e-shared-b-12efc5-serving-cert" not found Jul 16 05:12:08.282347 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.282294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9n\" (UniqueName: \"kubernetes.io/projected/77287066-a06b-4f29-941a-fe165f9ee72b-kube-api-access-48n9n\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.282347 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:12:08.282322 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77287066-a06b-4f29-941a-fe165f9ee72b-maas-api-tls podName:77287066-a06b-4f29-941a-fe165f9ee72b nodeName:}" failed. No retries permitted until 2026-07-16 05:12:08.782306343 +0000 UTC m=+1976.100354880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77287066-a06b-4f29-941a-fe165f9ee72b-maas-api-tls") pod "maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" (UID: "77287066-a06b-4f29-941a-fe165f9ee72b") : secret "maas-api-e2e-shared-b-12efc5-serving-cert" not found Jul 16 05:12:08.282827 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.282809 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77287066-a06b-4f29-941a-fe165f9ee72b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.293738 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.293704 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48n9n\" (UniqueName: \"kubernetes.io/projected/77287066-a06b-4f29-941a-fe165f9ee72b-kube-api-access-48n9n\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.785909 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.785877 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77287066-a06b-4f29-941a-fe165f9ee72b-maas-api-tls\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.788335 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.788313 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77287066-a06b-4f29-941a-fe165f9ee72b-maas-api-tls\") pod \"maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:08.977375 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:08.977339 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:09.115031 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:09.114929 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd"] Jul 16 05:12:09.117309 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:12:09.117283 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77287066_a06b_4f29_941a_fe165f9ee72b.slice/crio-6aadc056ed3a1131019f1592ae4df65d448079e78d9616f3072c82bb4f6f6472 WatchSource:0}: Error finding container 6aadc056ed3a1131019f1592ae4df65d448079e78d9616f3072c82bb4f6f6472: Status 404 returned error can't find the container with id 6aadc056ed3a1131019f1592ae4df65d448079e78d9616f3072c82bb4f6f6472 Jul 16 05:12:09.681823 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:09.681757 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" event={"ID":"77287066-a06b-4f29-941a-fe165f9ee72b","Type":"ContainerStarted","Data":"b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89"} Jul 16 05:12:09.682014 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:09.681886 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" event={"ID":"77287066-a06b-4f29-941a-fe165f9ee72b","Type":"ContainerStarted","Data":"6aadc056ed3a1131019f1592ae4df65d448079e78d9616f3072c82bb4f6f6472"} Jul 16 05:12:09.682014 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:09.681915 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:09.701292 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:09.701242 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" podStartSLOduration=1.34161889 podStartE2EDuration="1.701227858s" totalImagesPullingTime="359.608968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:12:09.698961715 +0000 UTC m=+1977.017010300" watchObservedRunningTime="2026-07-16 05:12:09.701227858 +0000 UTC m=+1977.019276417" Jul 16 05:12:15.691540 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:15.691512 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:12:15.893451 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:15.893421 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:12:30.983109 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:30.983071 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:12:52.382394 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:12:52.382351 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:13:06.589580 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:13:06.589543 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:13:21.373357 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:13:21.373312 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:13:38.681805 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:13:38.681710 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:13:54.384878 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:13:54.384849 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:14:12.685217 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:14:12.685180 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:14:13.263222 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:14:13.263191 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 05:14:13.266045 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:14:13.266020 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 05:14:13.270331 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:14:13.270309 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 05:14:13.272651 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:14:13.272635 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s27vv_1258efcc-2979-4c2a-a9a3-26f06d1699bd/ovn-acl-logging/0.log" Jul 16 05:14:29.777930 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:14:29.777891 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:14:44.585884 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:14:44.585847 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:15:00.168032 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.167998 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j"] Jul 16 05:15:00.171237 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.171221 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j" Jul 16 05:15:00.174397 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.174373 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7xp78\"" Jul 16 05:15:00.183113 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.183095 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j" Jul 16 05:15:00.191644 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.191620 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v"] Jul 16 05:15:00.195704 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.195486 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j"] Jul 16 05:15:00.195704 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.195584 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v" Jul 16 05:15:00.196283 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.196261 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84"] Jul 16 05:15:00.199509 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.199486 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84" Jul 16 05:15:00.218697 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.218636 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v"] Jul 16 05:15:00.220278 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.220224 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84"] Jul 16 05:15:00.223505 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.223486 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v" Jul 16 05:15:00.231782 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.231741 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84" Jul 16 05:15:00.392189 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.392160 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j"] Jul 16 05:15:00.393649 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:15:00.393614 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf37ba53_34ea_443a_938f_9d1428c3a1bd.slice/crio-907cb9ce5cdb16c204d38278ed6e0aa2e047749967559829cc659898d6d564d7 WatchSource:0}: Error finding container 907cb9ce5cdb16c204d38278ed6e0aa2e047749967559829cc659898d6d564d7: Status 404 returned error can't find the container with id 907cb9ce5cdb16c204d38278ed6e0aa2e047749967559829cc659898d6d564d7 Jul 16 05:15:00.418827 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.418803 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84"] Jul 16 05:15:00.427803 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:15:00.427755 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cf256e_9d97_4bd9_8bb5_2b5c43acc267.slice/crio-59ff4201ec411ecb4ceaa07319e5a3e879d984712046ac6ad507093a45240936 WatchSource:0}: Error finding container 59ff4201ec411ecb4ceaa07319e5a3e879d984712046ac6ad507093a45240936: Status 404 returned error can't find the container with id 59ff4201ec411ecb4ceaa07319e5a3e879d984712046ac6ad507093a45240936 Jul 16 05:15:00.434925 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.434903 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j" event={"ID":"cf37ba53-34ea-443a-938f-9d1428c3a1bd","Type":"ContainerStarted","Data":"907cb9ce5cdb16c204d38278ed6e0aa2e047749967559829cc659898d6d564d7"} Jul 16 05:15:00.436033 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.436010 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84" event={"ID":"24cf256e-9d97-4bd9-8bb5-2b5c43acc267","Type":"ContainerStarted","Data":"59ff4201ec411ecb4ceaa07319e5a3e879d984712046ac6ad507093a45240936"} Jul 16 05:15:00.593700 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:00.593675 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v"] Jul 16 05:15:00.596509 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:15:00.596483 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910e8f01_878a_4f20_8678_934783a3a12f.slice/crio-cf7f4a356c2a2dd747737dd2b6515af4fb05bd0e12a7ccbba655433d84e7420f WatchSource:0}: Error finding container cf7f4a356c2a2dd747737dd2b6515af4fb05bd0e12a7ccbba655433d84e7420f: Status 404 returned error can't find the container with id cf7f4a356c2a2dd747737dd2b6515af4fb05bd0e12a7ccbba655433d84e7420f Jul 16 05:15:01.441718 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:01.441686 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84" event={"ID":"24cf256e-9d97-4bd9-8bb5-2b5c43acc267","Type":"ContainerStarted","Data":"25e70ac1ba728724ff2f98e113a3e3b43b2990b632dd8080460b48fb985caa82"} Jul 16 05:15:01.443217 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:01.443184 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v" event={"ID":"910e8f01-878a-4f20-8678-934783a3a12f","Type":"ContainerStarted","Data":"81bfb1f7ff956270e3739b9ba0c066c65911d5a8668c693b402c385980696c46"} Jul 16 05:15:01.443360 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:01.443226 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v" event={"ID":"910e8f01-878a-4f20-8678-934783a3a12f","Type":"ContainerStarted","Data":"cf7f4a356c2a2dd747737dd2b6515af4fb05bd0e12a7ccbba655433d84e7420f"} Jul 16 05:15:01.444621 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:01.444594 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j" event={"ID":"cf37ba53-34ea-443a-938f-9d1428c3a1bd","Type":"ContainerStarted","Data":"cc7b059f20b731729a206f66b4c90c2bda0f88d5b06887739d1bce5bc18cd871"} Jul 16 05:15:01.455749 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:01.455700 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84" podStartSLOduration=1.455686443 podStartE2EDuration="1.455686443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:15:01.454114139 +0000 UTC m=+2148.772162697" watchObservedRunningTime="2026-07-16 05:15:01.455686443 +0000 UTC m=+2148.773734980" Jul 16 05:15:01.469696 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:01.469654 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v" podStartSLOduration=1.469645725 podStartE2EDuration="1.469645725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:15:01.467951808 +0000 UTC m=+2148.786000368" watchObservedRunningTime="2026-07-16 05:15:01.469645725 +0000 UTC m=+2148.787694284" Jul 16 05:15:01.483241 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:01.482982 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j" podStartSLOduration=1.482963449 podStartE2EDuration="1.482963449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:15:01.481322428 +0000 UTC m=+2148.799370988" watchObservedRunningTime="2026-07-16 05:15:01.482963449 +0000 UTC m=+2148.801012009" Jul 16 05:15:02.450145 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:02.450110 2585 generic.go:356] "Generic (PLEG): container finished" podID="cf37ba53-34ea-443a-938f-9d1428c3a1bd" containerID="cc7b059f20b731729a206f66b4c90c2bda0f88d5b06887739d1bce5bc18cd871" exitCode=0 Jul 16 05:15:02.450621 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:02.450200 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j" event={"ID":"cf37ba53-34ea-443a-938f-9d1428c3a1bd","Type":"ContainerDied","Data":"cc7b059f20b731729a206f66b4c90c2bda0f88d5b06887739d1bce5bc18cd871"} Jul 16 05:15:02.451627 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:02.451605 2585 generic.go:356] "Generic (PLEG): container finished" podID="24cf256e-9d97-4bd9-8bb5-2b5c43acc267" containerID="25e70ac1ba728724ff2f98e113a3e3b43b2990b632dd8080460b48fb985caa82" exitCode=0 Jul 16 05:15:02.451720 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:02.451691 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84" event={"ID":"24cf256e-9d97-4bd9-8bb5-2b5c43acc267","Type":"ContainerDied","Data":"25e70ac1ba728724ff2f98e113a3e3b43b2990b632dd8080460b48fb985caa82"} Jul 16 05:15:02.453032 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:02.453012 2585 generic.go:356] "Generic (PLEG): container finished" podID="910e8f01-878a-4f20-8678-934783a3a12f" containerID="81bfb1f7ff956270e3739b9ba0c066c65911d5a8668c693b402c385980696c46" exitCode=0 Jul 16 05:15:02.453145 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:02.453040 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v" event={"ID":"910e8f01-878a-4f20-8678-934783a3a12f","Type":"ContainerDied","Data":"81bfb1f7ff956270e3739b9ba0c066c65911d5a8668c693b402c385980696c46"} Jul 16 05:15:03.588065 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:03.588038 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84" Jul 16 05:15:03.643095 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:03.643069 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j" Jul 16 05:15:03.645983 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:03.645966 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v" Jul 16 05:15:04.276850 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.276818 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:15:04.463855 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.463820 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84" Jul 16 05:15:04.464054 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.463835 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84" event={"ID":"24cf256e-9d97-4bd9-8bb5-2b5c43acc267","Type":"ContainerDied","Data":"59ff4201ec411ecb4ceaa07319e5a3e879d984712046ac6ad507093a45240936"} Jul 16 05:15:04.464054 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.463986 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59ff4201ec411ecb4ceaa07319e5a3e879d984712046ac6ad507093a45240936" Jul 16 05:15:04.465260 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.465234 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v" Jul 16 05:15:04.465397 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.465279 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v" event={"ID":"910e8f01-878a-4f20-8678-934783a3a12f","Type":"ContainerDied","Data":"cf7f4a356c2a2dd747737dd2b6515af4fb05bd0e12a7ccbba655433d84e7420f"} Jul 16 05:15:04.465397 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.465335 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf7f4a356c2a2dd747737dd2b6515af4fb05bd0e12a7ccbba655433d84e7420f" Jul 16 05:15:04.466671 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.466649 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j" event={"ID":"cf37ba53-34ea-443a-938f-9d1428c3a1bd","Type":"ContainerDied","Data":"907cb9ce5cdb16c204d38278ed6e0aa2e047749967559829cc659898d6d564d7"} Jul 16 05:15:04.466849 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.466828 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="907cb9ce5cdb16c204d38278ed6e0aa2e047749967559829cc659898d6d564d7" Jul 16 05:15:04.466849 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:04.466668 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736315-t5z5j" Jul 16 05:15:26.903166 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:26.903080 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:15:41.875032 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:15:41.874992 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:16:00.976450 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:16:00.976415 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:16:19.576191 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:16:19.576158 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:16:47.985218 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:16:47.985186 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:17:02.472654 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:02.472616 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:17:17.884838 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:17.884799 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:17:33.281094 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:33.281060 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:17:51.398254 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.398219 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8"] Jul 16 05:17:51.398748 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.398405 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" podUID="ba53e3ad-216b-4347-ac5c-5816f378116f" containerName="maas-api" containerID="cri-o://77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011" gracePeriod=30 Jul 16 05:17:51.445503 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.445473 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84"] Jul 16 05:17:51.449874 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.449851 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a01a82-29736315-kpk84"] Jul 16 05:17:51.645332 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.645309 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:17:51.824410 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.824382 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba53e3ad-216b-4347-ac5c-5816f378116f-maas-api-tls\") pod \"ba53e3ad-216b-4347-ac5c-5816f378116f\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " Jul 16 05:17:51.824410 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.824420 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvkwv\" (UniqueName: \"kubernetes.io/projected/ba53e3ad-216b-4347-ac5c-5816f378116f-kube-api-access-nvkwv\") pod \"ba53e3ad-216b-4347-ac5c-5816f378116f\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " Jul 16 05:17:51.824674 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.824523 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53e3ad-216b-4347-ac5c-5816f378116f-openshift-service-ca\") pod \"ba53e3ad-216b-4347-ac5c-5816f378116f\" (UID: \"ba53e3ad-216b-4347-ac5c-5816f378116f\") " Jul 16 05:17:51.824929 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.824902 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba53e3ad-216b-4347-ac5c-5816f378116f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba53e3ad-216b-4347-ac5c-5816f378116f" (UID: "ba53e3ad-216b-4347-ac5c-5816f378116f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 05:17:51.826526 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.826501 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba53e3ad-216b-4347-ac5c-5816f378116f-kube-api-access-nvkwv" (OuterVolumeSpecName: "kube-api-access-nvkwv") pod "ba53e3ad-216b-4347-ac5c-5816f378116f" (UID: "ba53e3ad-216b-4347-ac5c-5816f378116f"). InnerVolumeSpecName "kube-api-access-nvkwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 05:17:51.826635 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.826583 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba53e3ad-216b-4347-ac5c-5816f378116f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba53e3ad-216b-4347-ac5c-5816f378116f" (UID: "ba53e3ad-216b-4347-ac5c-5816f378116f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 05:17:51.926006 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.925976 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba53e3ad-216b-4347-ac5c-5816f378116f-maas-api-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:17:51.926006 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.926002 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvkwv\" (UniqueName: \"kubernetes.io/projected/ba53e3ad-216b-4347-ac5c-5816f378116f-kube-api-access-nvkwv\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:17:51.926006 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:51.926012 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba53e3ad-216b-4347-ac5c-5816f378116f-openshift-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:17:52.221800 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:52.221678 2585 generic.go:356] "Generic (PLEG): container finished" podID="ba53e3ad-216b-4347-ac5c-5816f378116f" containerID="77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011" exitCode=0 Jul 16 05:17:52.221800 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:52.221734 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" Jul 16 05:17:52.221800 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:52.221774 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" event={"ID":"ba53e3ad-216b-4347-ac5c-5816f378116f","Type":"ContainerDied","Data":"77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011"} Jul 16 05:17:52.222083 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:52.221832 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8" event={"ID":"ba53e3ad-216b-4347-ac5c-5816f378116f","Type":"ContainerDied","Data":"3ff87c7234c6368ce144ae3b8166f42ad527ce58223c0859a1c0f1d983400582"} Jul 16 05:17:52.222083 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:52.221861 2585 scope.go:117] "RemoveContainer" containerID="77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011" Jul 16 05:17:52.231152 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:52.231134 2585 scope.go:117] "RemoveContainer" containerID="77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011" Jul 16 05:17:52.231411 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:17:52.231393 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011\": container with ID starting with 77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011 not found: ID does not exist" containerID="77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011" Jul 16 05:17:52.231453 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:52.231420 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011"} err="failed to get container status \"77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011\": rpc error: code = NotFound desc = could not find container \"77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011\": container with ID starting with 77356f3b45c8de5ae80da7c7f18a8ae387bb9e8d2f7c8bfde8839e1493a07011 not found: ID does not exist" Jul 16 05:17:52.250280 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:52.250249 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8"] Jul 16 05:17:52.252448 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:52.252427 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a01a82-cfbff4d6f-w7gl8"] Jul 16 05:17:53.208879 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:53.208849 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24cf256e-9d97-4bd9-8bb5-2b5c43acc267" path="/var/lib/kubelet/pods/24cf256e-9d97-4bd9-8bb5-2b5c43acc267/volumes" Jul 16 05:17:53.209255 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:53.209218 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba53e3ad-216b-4347-ac5c-5816f378116f" path="/var/lib/kubelet/pods/ba53e3ad-216b-4347-ac5c-5816f378116f/volumes" Jul 16 05:17:54.790197 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:17:54.790157 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:18:04.196091 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.196060 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd"] Jul 16 05:18:04.196706 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.196286 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" podUID="77287066-a06b-4f29-941a-fe165f9ee72b" containerName="maas-api" containerID="cri-o://b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89" gracePeriod=30 Jul 16 05:18:04.226721 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.226694 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v"] Jul 16 05:18:04.229679 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.229657 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-12efc5-29736315-9kr5v"] Jul 16 05:18:04.644954 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.644933 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:18:04.721085 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.721013 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77287066-a06b-4f29-941a-fe165f9ee72b-openshift-service-ca\") pod \"77287066-a06b-4f29-941a-fe165f9ee72b\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " Jul 16 05:18:04.721085 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.721077 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48n9n\" (UniqueName: \"kubernetes.io/projected/77287066-a06b-4f29-941a-fe165f9ee72b-kube-api-access-48n9n\") pod \"77287066-a06b-4f29-941a-fe165f9ee72b\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " Jul 16 05:18:04.721270 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.721132 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77287066-a06b-4f29-941a-fe165f9ee72b-maas-api-tls\") pod \"77287066-a06b-4f29-941a-fe165f9ee72b\" (UID: \"77287066-a06b-4f29-941a-fe165f9ee72b\") " Jul 16 05:18:04.721366 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.721345 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77287066-a06b-4f29-941a-fe165f9ee72b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77287066-a06b-4f29-941a-fe165f9ee72b" (UID: "77287066-a06b-4f29-941a-fe165f9ee72b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 05:18:04.723158 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.723130 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77287066-a06b-4f29-941a-fe165f9ee72b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77287066-a06b-4f29-941a-fe165f9ee72b" (UID: "77287066-a06b-4f29-941a-fe165f9ee72b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 05:18:04.723270 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.723174 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77287066-a06b-4f29-941a-fe165f9ee72b-kube-api-access-48n9n" (OuterVolumeSpecName: "kube-api-access-48n9n") pod "77287066-a06b-4f29-941a-fe165f9ee72b" (UID: "77287066-a06b-4f29-941a-fe165f9ee72b"). InnerVolumeSpecName "kube-api-access-48n9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 05:18:04.821878 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.821844 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48n9n\" (UniqueName: \"kubernetes.io/projected/77287066-a06b-4f29-941a-fe165f9ee72b-kube-api-access-48n9n\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:18:04.821878 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.821872 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77287066-a06b-4f29-941a-fe165f9ee72b-maas-api-tls\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:18:04.821878 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:04.821882 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77287066-a06b-4f29-941a-fe165f9ee72b-openshift-service-ca\") on node \"ip-10-0-140-3.ec2.internal\" DevicePath \"\"" Jul 16 05:18:05.209521 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.209468 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="910e8f01-878a-4f20-8678-934783a3a12f" path="/var/lib/kubelet/pods/910e8f01-878a-4f20-8678-934783a3a12f/volumes" Jul 16 05:18:05.283898 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.283866 2585 generic.go:356] "Generic (PLEG): container finished" podID="77287066-a06b-4f29-941a-fe165f9ee72b" containerID="b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89" exitCode=0 Jul 16 05:18:05.283898 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.283903 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" event={"ID":"77287066-a06b-4f29-941a-fe165f9ee72b","Type":"ContainerDied","Data":"b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89"} Jul 16 05:18:05.284147 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.283921 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" Jul 16 05:18:05.284147 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.283945 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd" event={"ID":"77287066-a06b-4f29-941a-fe165f9ee72b","Type":"ContainerDied","Data":"6aadc056ed3a1131019f1592ae4df65d448079e78d9616f3072c82bb4f6f6472"} Jul 16 05:18:05.284147 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.283970 2585 scope.go:117] "RemoveContainer" containerID="b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89" Jul 16 05:18:05.292607 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.292586 2585 scope.go:117] "RemoveContainer" containerID="b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89" Jul 16 05:18:05.292973 ip-10-0-140-3 kubenswrapper[2585]: E0716 05:18:05.292952 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89\": container with ID starting with b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89 not found: ID does not exist" containerID="b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89" Jul 16 05:18:05.293066 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.292984 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89"} err="failed to get container status \"b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89\": rpc error: code = NotFound desc = could not find container \"b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89\": container with ID starting with b2d1b117ffab1a7f4633cea2ba525d762d4e7c936d8aa6de50abd5a446a00f89 not found: ID does not exist" Jul 16 05:18:05.300735 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.300711 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd"] Jul 16 05:18:05.307343 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:05.307321 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-12efc5-65bd74976c-99tqd"] Jul 16 05:18:07.209549 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:07.209514 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77287066-a06b-4f29-941a-fe165f9ee72b" path="/var/lib/kubelet/pods/77287066-a06b-4f29-941a-fe165f9ee72b/volumes" Jul 16 05:18:07.675808 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:07.675760 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bfh4b"] Jul 16 05:18:17.671162 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:17.671133 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b4b7cb559-kc8fd_c472e523-ede4-4668-8be0-d08e19d38054/copy-manifests/0.log" Jul 16 05:18:17.677287 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:17.677264 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b4b7cb559-kc8fd_c472e523-ede4-4668-8be0-d08e19d38054/manager/0.log" Jul 16 05:18:18.409795 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:18.409751 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29736300-pbv9k_dfe06419-9d5c-4e34-a528-6116a85e9c46/cleanup/0.log" Jul 16 05:18:18.521530 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:18.521504 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29736315-t5z5j_cf37ba53-34ea-443a-938f-9d1428c3a1bd/cleanup/0.log" Jul 16 05:18:20.854549 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:20.854520 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-bfh4b_b1417baf-fcbb-4d81-9b92-53fd1b99417a/limitador/0.log" Jul 16 05:18:21.492802 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:21.492754 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-fwk28_f97db3db-1a6f-40e2-b957-2041ece56b7a/discovery/0.log" Jul 16 05:18:22.209777 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:22.209747 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b679c4ddd-d64dk_fcf7a78a-b00a-4419-923e-5cbde11f9f85/router/0.log" Jul 16 05:18:22.808053 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:22.808023 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-74mmp_e9223697-3fd0-4e67-851e-a7b1ef10d708/main/0.log" Jul 16 05:18:31.908868 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:31.908841 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l6277_2385ede2-8c97-424e-b560-19011a292b9e/global-pull-secret-syncer/0.log" Jul 16 05:18:32.053511 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:32.053478 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nnfvz_496d22a4-290a-4a92-9a34-b3d48db8a114/konnectivity-agent/0.log" Jul 16 05:18:32.187842 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:32.187746 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-3.ec2.internal_b2d100063295820fbe31810c8234ecb8/haproxy/0.log" Jul 16 05:18:37.673851 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:37.673818 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-bfh4b_b1417baf-fcbb-4d81-9b92-53fd1b99417a/limitador/0.log" Jul 16 05:18:39.369073 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.369005 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-g544c_be0c3d73-85fb-40a9-a2af-001e8231724a/cluster-monitoring-operator/0.log" Jul 16 05:18:39.530455 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.530422 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-924nw_2818bc5b-8877-4715-b054-3ad6065eb6ed/monitoring-plugin/0.log" Jul 16 05:18:39.559789 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.559737 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gn9lh_9800c6e8-c167-4cf7-9da2-c09d37fb2872/node-exporter/0.log" Jul 16 05:18:39.585908 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.585887 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gn9lh_9800c6e8-c167-4cf7-9da2-c09d37fb2872/kube-rbac-proxy/0.log" Jul 16 05:18:39.609889 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.609868 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gn9lh_9800c6e8-c167-4cf7-9da2-c09d37fb2872/init-textfile/0.log" Jul 16 05:18:39.876026 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.876001 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_212248e0-9ac7-4a1d-b190-9eebc8d3d481/prometheus/0.log" Jul 16 05:18:39.900387 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.900367 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_212248e0-9ac7-4a1d-b190-9eebc8d3d481/config-reloader/0.log" Jul 16 05:18:39.921730 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.921711 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_212248e0-9ac7-4a1d-b190-9eebc8d3d481/thanos-sidecar/0.log" Jul 16 05:18:39.943877 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.943861 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_212248e0-9ac7-4a1d-b190-9eebc8d3d481/kube-rbac-proxy-web/0.log" Jul 16 05:18:39.964703 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.964686 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_212248e0-9ac7-4a1d-b190-9eebc8d3d481/kube-rbac-proxy/0.log" Jul 16 05:18:39.987955 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:39.987927 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_212248e0-9ac7-4a1d-b190-9eebc8d3d481/kube-rbac-proxy-thanos/0.log" Jul 16 05:18:40.011354 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:40.011332 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_212248e0-9ac7-4a1d-b190-9eebc8d3d481/init-config-reloader/0.log" Jul 16 05:18:40.042979 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:40.042956 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-4s9jr_6544a76c-1633-4df6-a536-3e5318a41cc2/prometheus-operator/0.log" Jul 16 05:18:40.065964 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:40.065941 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-4s9jr_6544a76c-1633-4df6-a536-3e5318a41cc2/kube-rbac-proxy/0.log" Jul 16 05:18:41.991212 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991177 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4"] Jul 16 05:18:41.991560 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991524 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24cf256e-9d97-4bd9-8bb5-2b5c43acc267" containerName="cleanup" Jul 16 05:18:41.991560 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991538 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="24cf256e-9d97-4bd9-8bb5-2b5c43acc267" containerName="cleanup" Jul 16 05:18:41.991560 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991544 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf37ba53-34ea-443a-938f-9d1428c3a1bd" containerName="cleanup" Jul 16 05:18:41.991560 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991550 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf37ba53-34ea-443a-938f-9d1428c3a1bd" containerName="cleanup" Jul 16 05:18:41.991560 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991558 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="910e8f01-878a-4f20-8678-934783a3a12f" containerName="cleanup" Jul 16 05:18:41.991560 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991564 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="910e8f01-878a-4f20-8678-934783a3a12f" containerName="cleanup" Jul 16 05:18:41.991806 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991572 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77287066-a06b-4f29-941a-fe165f9ee72b" containerName="maas-api" Jul 16 05:18:41.991806 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991577 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77287066-a06b-4f29-941a-fe165f9ee72b" containerName="maas-api" Jul 16 05:18:41.991806 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991597 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba53e3ad-216b-4347-ac5c-5816f378116f" containerName="maas-api" Jul 16 05:18:41.991806 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991604 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba53e3ad-216b-4347-ac5c-5816f378116f" containerName="maas-api" Jul 16 05:18:41.991806 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991663 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="77287066-a06b-4f29-941a-fe165f9ee72b" containerName="maas-api" Jul 16 05:18:41.991806 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991675 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="910e8f01-878a-4f20-8678-934783a3a12f" containerName="cleanup" Jul 16 05:18:41.991806 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991683 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf37ba53-34ea-443a-938f-9d1428c3a1bd" containerName="cleanup" Jul 16 05:18:41.991806 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991691 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba53e3ad-216b-4347-ac5c-5816f378116f" containerName="maas-api" Jul 16 05:18:41.991806 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.991698 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="24cf256e-9d97-4bd9-8bb5-2b5c43acc267" containerName="cleanup" Jul 16 05:18:41.994851 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.994828 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:41.997314 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.997293 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5c2z4\"/\"openshift-service-ca.crt\"" Jul 16 05:18:41.998063 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.998046 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5c2z4\"/\"kube-root-ca.crt\"" Jul 16 05:18:41.998063 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:41.998046 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5c2z4\"/\"default-dockercfg-jr99c\"" Jul 16 05:18:42.002162 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.002140 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4"] Jul 16 05:18:42.152633 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.152602 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-proc\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.152633 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.152633 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-lib-modules\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.152868 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.152703 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-podres\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.152868 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.152739 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnzk6\" (UniqueName: \"kubernetes.io/projected/ea646654-6d01-4fca-be12-a240c0319f5a-kube-api-access-gnzk6\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.152868 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.152785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-sys\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.210933 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.210903 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/1.log" Jul 16 05:18:42.214563 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.214544 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mlq84_9870a155-1b41-4300-9007-f16f24f9c3cd/console-operator/2.log" Jul 16 05:18:42.253540 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.253475 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-podres\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.253540 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.253505 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzk6\" (UniqueName: \"kubernetes.io/projected/ea646654-6d01-4fca-be12-a240c0319f5a-kube-api-access-gnzk6\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.253540 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.253528 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-sys\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.253720 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.253582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-proc\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.253720 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.253601 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-lib-modules\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.253720 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.253627 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-podres\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.253720 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.253682 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-sys\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.253720 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.253682 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-proc\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.253926 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.253750 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea646654-6d01-4fca-be12-a240c0319f5a-lib-modules\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.261488 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.261466 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzk6\" (UniqueName: \"kubernetes.io/projected/ea646654-6d01-4fca-be12-a240c0319f5a-kube-api-access-gnzk6\") pod \"perf-node-gather-daemonset-q8qb4\" (UID: \"ea646654-6d01-4fca-be12-a240c0319f5a\") " pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.305127 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.305102 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:42.430282 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.430250 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4"] Jul 16 05:18:42.432505 ip-10-0-140-3 kubenswrapper[2585]: W0716 05:18:42.432464 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podea646654_6d01_4fca_be12_a240c0319f5a.slice/crio-f750d25f5f3e6ddd596e9dacffe182718625220c2c148c266089d3823a5ed132 WatchSource:0}: Error finding container f750d25f5f3e6ddd596e9dacffe182718625220c2c148c266089d3823a5ed132: Status 404 returned error can't find the container with id f750d25f5f3e6ddd596e9dacffe182718625220c2c148c266089d3823a5ed132 Jul 16 05:18:42.454428 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.454402 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" event={"ID":"ea646654-6d01-4fca-be12-a240c0319f5a","Type":"ContainerStarted","Data":"f750d25f5f3e6ddd596e9dacffe182718625220c2c148c266089d3823a5ed132"} Jul 16 05:18:42.809445 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:42.809417 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-z7gd5_81762e9a-efa8-4042-9f35-3458db40e2d8/download-server/0.log" Jul 16 05:18:43.461929 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:43.461895 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" event={"ID":"ea646654-6d01-4fca-be12-a240c0319f5a","Type":"ContainerStarted","Data":"e2ee445e498a2f8d2d338ad0142ec12f50d39f0d38e8ae2d928ffb6d7ad672bc"} Jul 16 05:18:43.462319 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:43.461956 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:43.479382 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:43.479335 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" podStartSLOduration=2.47932146 podStartE2EDuration="2.47932146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 05:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 05:18:43.47804902 +0000 UTC m=+2370.796097593" watchObservedRunningTime="2026-07-16 05:18:43.47932146 +0000 UTC m=+2370.797370018" Jul 16 05:18:44.641826 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:44.641796 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-757q2_a1c56c68-9f4a-445b-9739-c33a31e7def0/dns/0.log" Jul 16 05:18:44.663325 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:44.663300 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-757q2_a1c56c68-9f4a-445b-9739-c33a31e7def0/kube-rbac-proxy/0.log" Jul 16 05:18:44.783687 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:44.783664 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tqh7x_0e63d7c4-a323-4925-a1e2-06690436efe2/dns-node-resolver/0.log" Jul 16 05:18:45.368004 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:45.367971 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57b9c5599-rzggx_19716920-78d2-4a9c-a3ca-6f0474e9b664/registry/0.log" Jul 16 05:18:45.416259 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:45.416226 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cgjjz_69a01906-2a04-452a-b44e-9f6971572c5d/node-ca/0.log" Jul 16 05:18:46.559104 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:46.559072 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-fwk28_f97db3db-1a6f-40e2-b957-2041ece56b7a/discovery/0.log" Jul 16 05:18:46.809431 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:46.809302 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b679c4ddd-d64dk_fcf7a78a-b00a-4419-923e-5cbde11f9f85/router/0.log" Jul 16 05:18:47.524564 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:47.524530 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qpks8_f6bf91ab-7bc9-4ccf-b5ef-bfbfbfa6bf66/serve-healthcheck-canary/0.log" Jul 16 05:18:48.053499 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:48.053473 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-46bgf_fbebbdaf-4535-414a-8bce-42ca78476948/kube-rbac-proxy/0.log" Jul 16 05:18:48.075184 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:48.075161 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-46bgf_fbebbdaf-4535-414a-8bce-42ca78476948/exporter/0.log" Jul 16 05:18:48.097167 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:48.097146 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-46bgf_fbebbdaf-4535-414a-8bce-42ca78476948/extractor/0.log" Jul 16 05:18:49.475741 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:49.475715 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5c2z4/perf-node-gather-daemonset-q8qb4" Jul 16 05:18:50.751291 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:50.751263 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b4b7cb559-kc8fd_c472e523-ede4-4668-8be0-d08e19d38054/manager/0.log" Jul 16 05:18:50.772691 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:50.772665 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b4b7cb559-kc8fd_c472e523-ede4-4668-8be0-d08e19d38054/copy-manifests/0.log" Jul 16 05:18:52.316538 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:52.316508 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-546f8cbbbf-vqgtc_b2090408-5fb5-433d-acdf-dd062fb418a7/manager/0.log" Jul 16 05:18:52.732275 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:52.732191 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-fwk28_f97db3db-1a6f-40e2-b957-2041ece56b7a/discovery/0.log" Jul 16 05:18:52.883318 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:52.883288 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b679c4ddd-d64dk_fcf7a78a-b00a-4419-923e-5cbde11f9f85/router/0.log" Jul 16 05:18:54.214172 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:54.214126 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-fwk28_f97db3db-1a6f-40e2-b957-2041ece56b7a/discovery/0.log" Jul 16 05:18:54.368912 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:54.368883 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b679c4ddd-d64dk_fcf7a78a-b00a-4419-923e-5cbde11f9f85/router/0.log" Jul 16 05:18:58.735945 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:58.735886 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4wmg9_00a467c5-6e94-4450-b9f0-c84d4e1de57c/kube-multus/0.log" Jul 16 05:18:58.762094 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:58.762070 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-55n8f_48c4ca63-e0a4-4cb0-be52-2476b3758d98/kube-multus-additional-cni-plugins/0.log" Jul 16 05:18:58.783600 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:58.783576 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-55n8f_48c4ca63-e0a4-4cb0-be52-2476b3758d98/egress-router-binary-copy/0.log" Jul 16 05:18:58.804676 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:58.804647 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-55n8f_48c4ca63-e0a4-4cb0-be52-2476b3758d98/cni-plugins/0.log" Jul 16 05:18:58.825818 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:58.825794 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-55n8f_48c4ca63-e0a4-4cb0-be52-2476b3758d98/bond-cni-plugin/0.log" Jul 16 05:18:58.846676 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:58.846655 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-55n8f_48c4ca63-e0a4-4cb0-be52-2476b3758d98/routeoverride-cni/0.log" Jul 16 05:18:58.868293 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:58.868263 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-55n8f_48c4ca63-e0a4-4cb0-be52-2476b3758d98/whereabouts-cni-bincopy/0.log" Jul 16 05:18:58.890521 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:58.890504 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-55n8f_48c4ca63-e0a4-4cb0-be52-2476b3758d98/whereabouts-cni/0.log" Jul 16 05:18:59.405209 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:59.405185 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nsrxl_42f5e13d-20f3-46ed-ab64-13ce54e5e719/network-metrics-daemon/0.log" Jul 16 05:18:59.425877 ip-10-0-140-3 kubenswrapper[2585]: I0716 05:18:59.425853 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nsrxl_42f5e13d-20f3-46ed-ab64-13ce54e5e719/kube-rbac-proxy/0.log"